Oct 30 03:55:16.052915 ip-10-0-138-177 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 03:55:16.052925 ip-10-0-138-177 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 03:55:16.052934 ip-10-0-138-177 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 03:55:16.053235 ip-10-0-138-177 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 03:55:26.080025 ip-10-0-138-177 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 03:55:26.080043 ip-10-0-138-177 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fbbe49e26be64aff96b15679c120a83d -- Oct 30 03:57:16.366698 ip-10-0-138-177 systemd[1]: Starting Kubernetes Kubelet... Oct 30 03:57:16.779293 ip-10-0-138-177 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:57:16.779293 ip-10-0-138-177 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 03:57:16.779293 ip-10-0-138-177 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:57:16.779293 ip-10-0-138-177 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:57:16.785079 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785002 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:16.785079 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785077 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:16.785079 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785081 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:16.785079 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785084 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785087 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785091 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785094 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785097 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785099 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785102 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785105 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785108 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785111 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785114 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785116 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785119 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785122 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785124 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785126 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785131 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785135 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785138 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:16.785244 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785141 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785144 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785147 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785152 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785156 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785159 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785162 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785165 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785168 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785171 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785174 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785176 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785179 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785181 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785184 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785188 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785190 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785193 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785195 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:16.785700 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785198 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785201 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785204 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785206 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785209 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785211 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785214 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785216 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785219 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785229 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785232 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785235 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785238 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785240 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785243 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785245 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785249 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785251 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785254 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785257 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:16.786157 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785259 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785262 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785264 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785267 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785269 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785272 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785275 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785278 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785281 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785284 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785287 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785290 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785292 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785295 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785297 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785300 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785302 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785305 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785308 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:16.786643 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785310 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785313 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785316 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785325 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785327 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785330 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785332 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785335 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785337 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785340 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785343 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785345 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785348 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785350 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785353 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785357 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785360 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785363 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785365 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785368 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:16.787086 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785370 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785373 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785377 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785379 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785382 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.785384 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786160 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786168 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786171 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786174 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786176 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786179 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786181 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786184 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786187 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786189 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786193 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786195 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786198 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:16.787617 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786201 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786203 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786206 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786208 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786211 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786214 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786217 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786220 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786222 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786225 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786228 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786230 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786233 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786235 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786238 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786241 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786244 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786246 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786249 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:16.788090 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786251 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786254 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786257 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786259 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786262 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786265 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786267 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786270 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786272 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786275 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786277 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786282 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786284 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786287 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786289 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786292 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786295 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786297 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786300 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786302 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:16.788563 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786305 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786307 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786310 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786312 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786315 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786317 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786320 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786323 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786325 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786328 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786330 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786333 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786335 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786338 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786340 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786343 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786345 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786348 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786350 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786352 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:16.789047 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786355 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786357 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786360 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786362 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786366 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786369 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786371 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786374 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786376 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786379 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786382 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786384 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786387 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786390 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786392 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786395 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786397 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786400 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786402 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786405 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:16.789532 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786408 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786410 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786413 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786429 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786432 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786434 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786439 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786442 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786445 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786448 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786451 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786455 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786459 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786462 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786959 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786967 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786971 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786974 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786977 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:16.790005 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786979 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786982 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786985 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786987 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786990 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786993 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786995 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.786998 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787000 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787003 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787005 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787008 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787011 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787014 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787016 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787019 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787022 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787024 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787027 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787029 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:16.790466 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787032 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787035 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787037 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787040 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787042 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787045 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787047 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787050 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787054 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787057 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787060 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787063 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787065 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787068 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787070 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787073 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787075 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787078 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787081 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:16.790948 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787083 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787087 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787089 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787092 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787095 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787098 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787101 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787103 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787106 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787109 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787112 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787114 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787116 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787119 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787121 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787124 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787127 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787129 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787131 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:16.791447 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787134 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787137 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787141 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787143 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787146 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787149 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787152 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787154 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787157 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787159 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787162 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787164 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787167 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787171 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787175 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787178 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787181 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787185 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787191 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:16.791913 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787193 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787196 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787199 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787201 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787203 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787206 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787208 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787211 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787214 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787216 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787218 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787221 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787224 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787226 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787229 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787232 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787235 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787237 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787240 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:16.792387 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787243 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787248 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787251 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787254 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.787256 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787354 2570 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787361 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787367 2570 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787372 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787377 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787380 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787385 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787390 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787394 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787397 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787400 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787404 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787407 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787410 2570 flags.go:64] FLAG: --cgroup-root="" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787413 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787429 2570 flags.go:64] FLAG: --client-ca-file="" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787432 2570 flags.go:64] FLAG: --cloud-provider="external" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787435 2570 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 03:57:16.792865 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787440 2570 flags.go:64] FLAG: --cluster-domain="" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787443 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787446 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787449 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787452 2570 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787457 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787460 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787463 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787467 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787470 2570 flags.go:64] FLAG: --contention-profiling="false" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787473 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787476 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787480 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787483 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787487 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787490 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787493 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787496 2570 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787499 2570 flags.go:64] FLAG: --enable-server="true" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787502 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787506 2570 flags.go:64] FLAG: --event-burst="100" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787510 2570 flags.go:64] FLAG: --event-qps="50" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787513 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787516 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 03:57:16.793446 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787519 2570 flags.go:64] FLAG: --eviction-hard="" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787523 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787526 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787529 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787532 2570 flags.go:64] FLAG: --eviction-soft="" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787535 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787538 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787541 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787544 2570 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787547 2570 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787550 2570 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787553 2570 flags.go:64] FLAG: --feature-gates="" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787557 2570 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787561 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787564 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787568 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787571 2570 flags.go:64] FLAG: --healthz-port="10248" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787574 2570 flags.go:64] FLAG: --help="false" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787577 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-138-177.ec2.internal" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787581 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787585 2570 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787587 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787591 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 03:57:16.794050 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787594 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787597 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787600 2570 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787603 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787606 2570 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787609 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787614 2570 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787617 2570 flags.go:64] FLAG: --kube-reserved="" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787620 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787623 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787626 2570 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787629 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787632 2570 flags.go:64] FLAG: --lock-file="" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787635 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787638 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787641 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787646 2570 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787649 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787652 2570 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787655 2570 flags.go:64] FLAG: --logging-format="text" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787658 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787662 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787666 2570 flags.go:64] FLAG: --manifest-url="" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787669 2570 flags.go:64] FLAG: --manifest-url-header="" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787674 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 03:57:16.794610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787677 2570 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787681 2570 flags.go:64] FLAG: --max-pods="110" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787684 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787687 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787691 2570 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787693 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787696 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787699 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787702 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787716 2570 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787719 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787722 2570 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787725 2570 flags.go:64] FLAG: --pod-cidr="" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787729 2570 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787732 2570 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787735 2570 flags.go:64] FLAG: --pods-per-core="0" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787738 2570 flags.go:64] FLAG: --port="10250" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787741 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787744 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01e5b2c5e304b54b7" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787747 2570 flags.go:64] FLAG: --qos-reserved="" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787750 2570 flags.go:64] FLAG: --read-only-port="10255" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787753 2570 flags.go:64] FLAG: --register-node="true" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787756 2570 flags.go:64] FLAG: --register-with-taints="" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787760 2570 flags.go:64] FLAG: --registry-burst="10" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787763 2570 flags.go:64] FLAG: --registry-qps="5" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787766 2570 flags.go:64] FLAG: --reserved-cpus="" Oct 30 03:57:16.795218 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787769 2570 flags.go:64] FLAG: --reserved-memory="" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787773 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787776 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787780 2570 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787783 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787786 2570 flags.go:64] FLAG: --runonce="false" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787789 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787792 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787795 2570 flags.go:64] FLAG: --seccomp-default="false" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787798 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787801 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787804 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787807 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787810 2570 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787813 2570 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787816 2570 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787819 2570 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787822 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787825 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787829 2570 flags.go:64] FLAG: --system-cgroups="" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787832 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787845 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787848 2570 flags.go:64] FLAG: --tls-cert-file="" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787851 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787856 2570 flags.go:64] FLAG: --tls-min-version="" Oct 30 03:57:16.795959 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787859 2570 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787862 2570 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787865 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787868 2570 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787871 2570 flags.go:64] FLAG: --v="2" Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787876 2570 flags.go:64] FLAG: --version="false" Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787880 2570 flags.go:64] FLAG: --vmodule="" Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787883 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.787887 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788017 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788021 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788024 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788028 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788030 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788033 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788036 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788038 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788041 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788044 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788047 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788050 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788052 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:16.796558 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788055 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788057 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788060 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788062 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788066 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788069 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788071 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788074 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788076 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788079 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788081 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788084 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788087 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788089 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788092 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788095 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788097 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788100 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788103 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788105 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:16.797071 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788109 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788111 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788114 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788116 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788119 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788121 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788124 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788127 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788130 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788132 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788135 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788138 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788140 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788143 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788145 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788148 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788152 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788154 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788157 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788159 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:16.797553 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788162 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788164 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788167 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788169 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788172 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788174 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788177 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788181 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788184 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788187 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788189 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788192 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788194 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788197 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788199 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788202 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788205 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788207 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788210 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:16.798027 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788213 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788216 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788220 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788223 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788228 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788231 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788233 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788236 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788238 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788242 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788245 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788247 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788250 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788253 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788255 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788258 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788260 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788263 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788266 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:16.798575 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788268 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788271 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788274 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788276 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788279 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788281 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788284 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788287 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788289 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788292 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788294 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788297 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788299 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788302 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788304 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.788329 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:57:16.799040 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788365 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788370 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788374 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788377 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788380 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788383 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788385 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788389 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788392 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788394 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788397 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788400 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788402 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788405 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788407 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788410 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788413 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788426 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788429 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788431 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:16.799433 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788434 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788437 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788440 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788442 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788445 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788447 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788450 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788452 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788455 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788458 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788460 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788463 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788466 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788469 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788472 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788475 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788477 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788480 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788482 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:16.799931 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788486 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788489 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788493 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788496 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788499 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788503 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788506 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788508 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788513 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788516 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788519 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788522 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788525 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788528 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788531 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788535 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788538 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788542 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788544 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:16.800398 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788548 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788550 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788554 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788556 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788559 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788561 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788564 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788567 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788570 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788572 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788576 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788579 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788581 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788586 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788588 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788591 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788593 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788596 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788599 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788601 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:16.800897 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788604 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788608 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788610 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788613 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788616 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788619 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788622 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788624 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788627 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788629 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788632 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788634 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788637 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788640 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788643 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788646 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788648 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788651 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788653 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788656 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:16.801392 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788659 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788661 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788664 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788666 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788669 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788673 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788675 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.788678 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.795586 2570 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.795603 2570 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.795618 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.795622 2570 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795685 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795691 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795694 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795706 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795709 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795712 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795715 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:16.801879 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795718 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795721 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795723 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795726 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795729 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795732 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795735 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795737 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795740 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795743 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795746 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795748 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795751 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795754 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795756 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795759 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795761 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795764 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795767 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795769 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:16.802345 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795772 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795775 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795779 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795782 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795784 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795787 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795789 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795792 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795795 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795797 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795800 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795802 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795805 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795808 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795811 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795813 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795816 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795819 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795821 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:16.802826 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795824 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795827 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795830 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795832 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795835 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795868 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795872 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795875 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795877 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795880 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795883 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795886 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795888 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795891 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795893 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795896 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795900 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795903 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795906 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795908 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:16.803274 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795912 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795916 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795920 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795924 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795926 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795929 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795932 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795936 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795940 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795943 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795946 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795948 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795951 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795954 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795957 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795959 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795962 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795965 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795968 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:16.803769 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795971 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795974 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795976 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795979 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795981 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795984 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795987 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795989 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795992 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795995 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.795999 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796002 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796004 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796007 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796009 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796012 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796015 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796017 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796020 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796022 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:16.804255 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796025 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.796053 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796212 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796216 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796219 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796222 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796225 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796227 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796230 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796232 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796235 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796238 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796240 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796243 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796245 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796248 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:16.804729 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796251 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796253 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796256 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796258 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796262 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796264 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796267 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796270 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796274 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796278 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796282 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796286 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796289 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796292 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796295 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796298 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796301 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796304 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796306 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:16.805208 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796309 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796311 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796314 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796316 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796319 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796322 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796324 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796327 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796330 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796332 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796335 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796337 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796340 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796342 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796345 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796347 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796350 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796353 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796356 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796358 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:16.805929 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796361 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796364 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796367 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796369 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796372 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796374 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796377 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796380 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796382 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796384 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796387 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796390 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796392 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796395 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796397 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796400 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796402 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796405 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796407 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:16.806574 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796410 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796412 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796427 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796430 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796434 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796436 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796439 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796442 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796444 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796447 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796450 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796457 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796460 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796462 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796465 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796468 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796471 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796474 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796477 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796479 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:16.807067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796482 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796485 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796487 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796490 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796493 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796495 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796498 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796501 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796503 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796506 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796508 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796511 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796513 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796516 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.796542 2570 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796578 2570 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:16.807576 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796582 2570 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796585 2570 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796588 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796590 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796593 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796595 2570 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796598 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796600 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796604 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796607 2570 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796610 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796613 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796616 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796619 2570 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796621 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796624 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796626 2570 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796629 2570 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796632 2570 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:16.807990 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796634 2570 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796637 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796641 2570 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796645 2570 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796649 2570 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796652 2570 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796655 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796658 2570 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796660 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796663 2570 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796666 2570 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796668 2570 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796671 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796673 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796676 2570 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796679 2570 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796681 2570 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796684 2570 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796686 2570 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:16.808618 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796688 2570 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796691 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796695 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796698 2570 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796700 2570 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796703 2570 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796705 2570 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796708 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796711 2570 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796714 2570 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796716 2570 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796719 2570 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796721 2570 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796724 2570 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796727 2570 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796729 2570 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796732 2570 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796734 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796737 2570 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796740 2570 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:16.809078 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796743 2570 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796745 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796748 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796754 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796756 2570 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796759 2570 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796762 2570 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796764 2570 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796766 2570 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796769 2570 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796771 2570 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796774 2570 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796776 2570 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796779 2570 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796783 2570 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796785 2570 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796788 2570 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796790 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796793 2570 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:16.809560 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796795 2570 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796798 2570 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796801 2570 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796804 2570 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796806 2570 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796809 2570 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796812 2570 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796815 2570 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796817 2570 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796820 2570 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796822 2570 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796825 2570 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796828 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796830 2570 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796833 2570 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796835 2570 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796841 2570 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796844 2570 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796846 2570 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796849 2570 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:16.810067 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796852 2570 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796854 2570 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796857 2570 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796859 2570 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796862 2570 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796864 2570 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796867 2570 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:16.796870 2570 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.796923 2570 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.798796 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.799650 2570 server.go:1014] "Starting client certificate rotation" Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.799748 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.799784 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.802192 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:57:16.810581 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.803350 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:57:16.816076 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.816060 2570 log.go:25] "Validated CRI v1 runtime API" Oct 30 03:57:16.819935 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.819922 2570 log.go:25] "Validated CRI v1 image API" Oct 30 03:57:16.820753 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.820727 2570 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 03:57:16.823103 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.823080 2570 fs.go:135] Filesystem UUIDs: map[6289d932-d5da-45bf-bb60-ee9eb419b2ba:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9e810927-6396-4172-a431-27de3cc6f8e1:/dev/nvme0n1p3] Oct 30 03:57:16.823151 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.823103 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 03:57:16.829758 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.829650 2570 manager.go:217] Machine: {Timestamp:2026-10-30 03:57:16.827245333 +0000 UTC m=+0.339908991 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099562 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f587eec2a204283236b8693754f2 SystemUUID:ec22f587-eec2-a204-2832-36b8693754f2 BootID:fbbe49e2-6be6-4aff-96b1-5679c120a83d Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c2:ed:81:29 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c2:ed:81:29 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:8e:c4:ab:8f:65 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 03:57:16.829758 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.829756 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 03:57:16.829864 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.829830 2570 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 03:57:16.832816 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.832796 2570 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 03:57:16.832940 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.832819 2570 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-177.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 03:57:16.832980 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.832941 2570 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 03:57:16.832980 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.832949 2570 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 03:57:16.833038 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.832985 2570 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 03:57:16.833038 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.833003 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 03:57:16.833825 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.833814 2570 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 03:57:16.837487 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.837476 2570 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 03:57:16.837610 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.837601 2570 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 03:57:16.838306 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.838296 2570 kubelet.go:493] "Attempting to sync node with API server" Oct 30 03:57:16.838335 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.838311 2570 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 03:57:16.838335 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.838322 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 03:57:16.838335 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.838330 2570 kubelet.go:405] "Adding apiserver pod source" Oct 30 03:57:16.838436 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.838338 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 03:57:16.838661 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.838650 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 03:57:16.841263 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.841241 2570 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 03:57:16.845721 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.845709 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 03:57:16.846857 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.846845 2570 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 03:57:16.846909 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.846900 2570 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 03:57:16.847882 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.847870 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 03:57:16.847924 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.847890 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 03:57:16.847924 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.847897 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 03:57:16.847924 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.847903 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 03:57:16.847924 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.847909 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 03:57:16.847924 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.847915 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 03:57:16.847924 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.847923 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 03:57:16.848071 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.847929 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 03:57:16.848071 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.847936 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 03:57:16.848071 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.847941 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 03:57:16.848071 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.847957 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 03:57:16.848071 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.847965 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 03:57:16.848597 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.848588 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 03:57:16.848628 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.848598 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 03:57:16.852982 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.852965 2570 server.go:1263] "Started kubelet" Oct 30 03:57:16.853232 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.853064 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 03:57:16.853323 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.853249 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 03:57:16.853323 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.853180 2570 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 03:57:16.853755 ip-10-0-138-177 systemd[1]: Started Kubernetes Kubelet. Oct 30 03:57:16.854375 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.854361 2570 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 03:57:16.861277 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.861256 2570 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 03:57:16.861668 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.861653 2570 server.go:317] "Adding debug handlers to kubelet server" Oct 30 03:57:16.863617 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.863599 2570 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 03:57:16.863755 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.863612 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 03:57:16.863755 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.863690 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:16.864157 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.864136 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 03:57:16.864157 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.864140 2570 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 03:57:16.864291 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.864171 2570 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 03:57:16.864291 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.864261 2570 reconstruct.go:97] "Volume reconstruction finished" Oct 30 03:57:16.864291 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.864281 2570 reconciler.go:29] "Reconciler: start to sync state" Oct 30 03:57:16.864800 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.864530 2570 factory.go:55] Registering systemd factory Oct 30 03:57:16.864800 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.864549 2570 factory.go:223] Registration of the systemd container factory successfully Oct 30 03:57:16.864984 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.864800 2570 factory.go:153] Registering CRI-O factory Oct 30 03:57:16.864984 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.864814 2570 factory.go:223] Registration of the crio container factory successfully Oct 30 03:57:16.864984 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.864877 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 03:57:16.864984 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.864897 2570 factory.go:103] Registering Raw factory Oct 30 03:57:16.864984 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.864913 2570 manager.go:1199] Started watching for new ooms in manager Oct 30 03:57:16.865797 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.865782 2570 manager.go:319] Starting recovery of all containers Oct 30 03:57:16.876004 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.875981 2570 manager.go:324] Recovery completed Oct 30 03:57:16.877038 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.877014 2570 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 03:57:16.881373 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.881353 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:57:16.883528 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.883498 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:57:16.883599 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.883544 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:57:16.883599 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.883561 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:57:16.884088 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.884076 2570 cpu_manager.go:226] "Starting" policy="none" Oct 30 03:57:16.884088 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.884085 2570 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 03:57:16.884178 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.884089 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:57:16.884178 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.884100 2570 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 03:57:16.884805 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.884785 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:57:16.884871 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.884816 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:57:16.884871 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.884829 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:57:16.886243 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.886228 2570 policy_none.go:50] "Start" Oct 30 03:57:16.886243 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.886244 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 03:57:16.886324 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.886254 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 03:57:16.887476 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.887456 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-177.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 03:57:16.887476 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.887464 2570 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 03:57:16.888250 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.888237 2570 policy_none.go:44] "Start" Oct 30 03:57:16.888529 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.887463 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-177.ec2.internal.18e332881fbbeab9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-177.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-177.ec2.internal,},FirstTimestamp:2026-10-30 03:57:16.852939449 +0000 UTC m=+0.365603105,LastTimestamp:2026-10-30 03:57:16.852939449 +0000 UTC m=+0.365603105,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-177.ec2.internal,}" Oct 30 03:57:16.908656 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.908636 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 03:57:16.908777 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.908757 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-86h89" Oct 30 03:57:16.909610 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.909584 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 03:57:16.909695 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.909569 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-177.ec2.internal.18e33288218ea551 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-177.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-177.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-177.ec2.internal,},FirstTimestamp:2026-10-30 03:57:16.883526993 +0000 UTC m=+0.396190659,LastTimestamp:2026-10-30 03:57:16.883526993 +0000 UTC m=+0.396190659,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-177.ec2.internal,}" Oct 30 03:57:16.909778 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.909729 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 03:57:16.909832 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.909790 2570 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.920071 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-86h89" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.927993 2570 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.931146 2570 manager.go:341] "Starting Device Plugin manager" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.931167 2570 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.931175 2570 server.go:87] "Starting device plugin registration server" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.931388 2570 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.931398 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.931521 2570 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.931607 2570 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.931617 2570 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.932293 2570 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.932338 2570 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.932680 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.933892 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.933913 2570 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.933931 2570 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.933969 2570 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 03:57:16.941776 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:16.934045 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-177.ec2.internal.18e33288218f087e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-177.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-177.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-177.ec2.internal,},FirstTimestamp:2026-10-30 03:57:16.883552382 +0000 UTC m=+0.396216043,LastTimestamp:2026-10-30 03:57:16.883552382 +0000 UTC m=+0.396216043,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-177.ec2.internal,}" Oct 30 03:57:16.962149 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:16.962132 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:57:17.031718 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.031669 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:57:17.032526 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.032390 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:57:17.032612 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.032544 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:57:17.032612 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.032559 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:57:17.032612 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.032592 2570 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.034551 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.034521 2570 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal"] Oct 30 03:57:17.034613 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.034585 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:57:17.035258 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.035243 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:57:17.035328 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.035271 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:57:17.035328 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.035280 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:57:17.037450 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.037438 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:57:17.037597 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.037584 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.037636 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.037614 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:57:17.038122 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.038099 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:57:17.038122 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.038127 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:57:17.038254 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.038138 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:57:17.038254 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.038101 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:57:17.038254 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.038181 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:57:17.038254 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.038193 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:57:17.040252 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.040230 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.040316 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.040266 2570 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:57:17.040962 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.040947 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:57:17.041037 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.040971 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:57:17.041037 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.040980 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:57:17.053643 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.053625 2570 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.053643 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:17.053643 2570 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-177.ec2.internal\": node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:17.065969 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.065945 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a1a550d76a223b097bf2829db01e97d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" (UID: \"3a1a550d76a223b097bf2829db01e97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.065969 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.065969 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a1a550d76a223b097bf2829db01e97d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" (UID: \"3a1a550d76a223b097bf2829db01e97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.066120 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.065983 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/552cd6bdf9ef61710ec1ae6f1eae6b32-config\") pod \"kube-apiserver-proxy-ip-10-0-138-177.ec2.internal\" (UID: \"552cd6bdf9ef61710ec1ae6f1eae6b32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.069751 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:17.069735 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-177.ec2.internal\" not found" node="ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.072730 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:17.072710 2570 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-177.ec2.internal\" not found" node="ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.102523 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:17.102505 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:17.166226 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.166208 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a1a550d76a223b097bf2829db01e97d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" (UID: \"3a1a550d76a223b097bf2829db01e97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.166283 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.166231 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a1a550d76a223b097bf2829db01e97d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" (UID: \"3a1a550d76a223b097bf2829db01e97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.166283 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.166254 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/552cd6bdf9ef61710ec1ae6f1eae6b32-config\") pod \"kube-apiserver-proxy-ip-10-0-138-177.ec2.internal\" (UID: \"552cd6bdf9ef61710ec1ae6f1eae6b32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.166346 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.166299 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/552cd6bdf9ef61710ec1ae6f1eae6b32-config\") pod \"kube-apiserver-proxy-ip-10-0-138-177.ec2.internal\" (UID: \"552cd6bdf9ef61710ec1ae6f1eae6b32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.166346 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.166318 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a1a550d76a223b097bf2829db01e97d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" (UID: \"3a1a550d76a223b097bf2829db01e97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.166346 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.166318 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a1a550d76a223b097bf2829db01e97d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" (UID: \"3a1a550d76a223b097bf2829db01e97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.202575 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:17.202556 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:17.302655 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:17.302597 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:17.373489 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.373470 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.376060 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.376044 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:17.403190 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:17.403172 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:17.503981 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:17.503961 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:17.604491 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:17.604449 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:17.704902 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:17.704880 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:17.799406 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.799379 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 03:57:17.799827 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.799534 2570 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 03:57:17.805668 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:17.805648 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:17.893475 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:17.893449 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1a550d76a223b097bf2829db01e97d.slice/crio-2c0ddd85ce8284af9f2e2b438c86be40ea67764495e5ceaa1fab27d8513cf984 WatchSource:0}: Error finding container 2c0ddd85ce8284af9f2e2b438c86be40ea67764495e5ceaa1fab27d8513cf984: Status 404 returned error can't find the container with id 2c0ddd85ce8284af9f2e2b438c86be40ea67764495e5ceaa1fab27d8513cf984 Oct 30 03:57:17.898212 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.898198 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:57:17.905724 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:17.905704 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:17.916127 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:17.916108 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552cd6bdf9ef61710ec1ae6f1eae6b32.slice/crio-b53ba5682503e82dce02a1cf8ab9edb1e81f2ca6125a5f2e139764f496e5bd14 WatchSource:0}: Error finding container b53ba5682503e82dce02a1cf8ab9edb1e81f2ca6125a5f2e139764f496e5bd14: Status 404 returned error can't find the container with id b53ba5682503e82dce02a1cf8ab9edb1e81f2ca6125a5f2e139764f496e5bd14 Oct 30 03:57:17.925996 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.925968 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 03:52:16 +0000 UTC" deadline="2028-03-28 16:44:40.245182404 +0000 UTC" Oct 30 03:57:17.925996 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.925994 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12372h47m22.319191695s" Oct 30 03:57:17.937901 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.937864 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" event={"ID":"552cd6bdf9ef61710ec1ae6f1eae6b32","Type":"ContainerStarted","Data":"b53ba5682503e82dce02a1cf8ab9edb1e81f2ca6125a5f2e139764f496e5bd14"} Oct 30 03:57:17.939027 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.939006 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" event={"ID":"3a1a550d76a223b097bf2829db01e97d","Type":"ContainerStarted","Data":"2c0ddd85ce8284af9f2e2b438c86be40ea67764495e5ceaa1fab27d8513cf984"} Oct 30 03:57:17.964173 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:17.964156 2570 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:57:18.005962 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:18.005941 2570 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:18.015913 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.015890 2570 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:57:18.048990 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.048972 2570 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:57:18.064595 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.064575 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:18.088044 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.088025 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:57:18.090569 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.090556 2570 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:18.111742 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.111725 2570 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:57:18.839103 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.839076 2570 apiserver.go:52] "Watching apiserver" Oct 30 03:57:18.864896 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.864871 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 03:57:18.881095 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.881066 2570 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 03:57:18.881449 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.881401 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-x58t9","openshift-multus/network-metrics-daemon-tffth","openshift-network-diagnostics/network-check-target-v5zp7","openshift-network-operator/iptables-alerter-q98qf","kube-system/global-pull-secret-syncer-p2mmh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x","openshift-cluster-node-tuning-operator/tuned-dmdgx","openshift-dns/node-resolver-zjzzn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal","openshift-multus/multus-9fmv2","kube-system/konnectivity-agent-9gghj","openshift-ovn-kubernetes/ovnkube-node-krggs","kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal","openshift-multus/multus-additional-cni-plugins-lpm6b"] Oct 30 03:57:18.884108 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.884085 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.886205 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.886183 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:18.888296 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.888275 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:18.888397 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:18.888350 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:18.890487 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.890469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:18.892639 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.892616 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:18.892720 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:18.892678 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:18.895093 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.894992 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:18.897041 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.897021 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:18.899326 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.899306 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:57:18.899563 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.899546 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 03:57:18.899656 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.899630 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-skmdf\"" Oct 30 03:57:18.900580 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.900518 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:18.900985 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.900966 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 03:57:18.901078 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.901011 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dnvhh\"" Oct 30 03:57:18.901137 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.901086 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 03:57:18.902671 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.902652 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:18.902765 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:18.902723 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:18.904968 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.904949 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.907223 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.907202 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:18.915655 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.915634 2570 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 03:57:18.915755 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.915694 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qhk9w" Oct 30 03:57:18.915755 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.915735 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.924250 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.923430 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qhk9w" Oct 30 03:57:18.965587 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.965569 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 03:57:18.977516 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977496 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-system-cni-dir\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:18.977626 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977529 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-kubelet\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.977626 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977559 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-run-netns\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.977747 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977648 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-ovn\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.977747 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977685 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-ovnkube-script-lib\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.977747 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977712 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62ebb448-14fe-4919-9686-018754ede9cf-ovn-node-metrics-cert\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.977747 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977737 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-modprobe-d\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.977953 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977768 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-system-cni-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.977953 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977793 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-netns\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.977953 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977817 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-kubelet\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.977953 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977843 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-daemon-config\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.977953 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977877 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-multus-certs\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.977953 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977901 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.977953 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977924 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73ae3560-963a-46a6-a052-21db48a98360-host\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:18.978288 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977948 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-etc-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.978288 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.977982 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.978288 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-tuned\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.978288 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978035 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:18.978288 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978062 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twrkf\" (UniqueName: \"kubernetes.io/projected/471b293f-bc2d-4a9e-bddb-aea255ac235e-kube-api-access-twrkf\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:18.978288 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978086 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnbqs\" (UniqueName: \"kubernetes.io/projected/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-kube-api-access-jnbqs\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:18.978288 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978114 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sj9m\" (UniqueName: \"kubernetes.io/projected/91502bdf-1f01-469f-ac9f-64aa13e5563a-kube-api-access-7sj9m\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.978288 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978149 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysctl-d\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.978288 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978171 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-sys\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.978288 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978216 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-var-lib-kubelet\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.978288 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978241 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd9d674b-c3d9-4e78-bc55-81cedfc32881-konnectivity-ca\") pod \"konnectivity-agent-9gghj\" (UID: \"cd9d674b-c3d9-4e78-bc55-81cedfc32881\") " pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:18.978288 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978267 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-hostroot\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978296 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-systemd-units\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978318 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-log-socket\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978342 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwzs8\" (UniqueName: \"kubernetes.io/projected/62ebb448-14fe-4919-9686-018754ede9cf-kube-api-access-jwzs8\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978364 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-systemd\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978387 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-cnibin\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978410 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-slash\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978463 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978488 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-env-overrides\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978513 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrmh\" (UniqueName: \"kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh\") pod \"network-check-target-v5zp7\" (UID: \"438677a3-904a-474e-bbed-4032260a3a3c\") " pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978547 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4m9\" (UniqueName: \"kubernetes.io/projected/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-kube-api-access-dc4m9\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978572 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978598 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-device-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978625 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-hosts-file\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978648 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-cnibin\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978673 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-cni-bin\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.978851 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978697 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-systemd\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978722 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-node-log\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978745 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-run\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978768 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-lib-modules\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978793 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysctl-conf\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978818 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-host\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978842 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93cbeaeb-857f-4c3f-825f-62c46320edd0-dbus\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978866 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-registration-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978890 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-sys-fs\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978916 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-etc-kubernetes\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978946 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnjc9\" (UniqueName: \"kubernetes.io/projected/73ae3560-963a-46a6-a052-21db48a98360-kube-api-access-bnjc9\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978971 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-tmp\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.978995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979018 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-tmp-dir\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979043 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-host-slash\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979067 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-conf-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979091 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-cni-bin\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.979615 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysconfig\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979150 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d97vg\" (UniqueName: \"kubernetes.io/projected/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-kube-api-access-d97vg\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979176 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979202 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979226 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-socket-dir-parent\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979251 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-cni-multus\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979275 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73ae3560-963a-46a6-a052-21db48a98360-serviceca\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979299 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd9d674b-c3d9-4e78-bc55-81cedfc32881-agent-certs\") pod \"konnectivity-agent-9gghj\" (UID: \"cd9d674b-c3d9-4e78-bc55-81cedfc32881\") " pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979326 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979351 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-var-lib-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979375 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-cni-netd\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979403 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979452 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-cni-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979574 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91502bdf-1f01-469f-ac9f-64aa13e5563a-cni-binary-copy\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979598 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-ovnkube-config\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979619 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-iptables-alerter-script\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:18.980354 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979636 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93cbeaeb-857f-4c3f-825f-62c46320edd0-kubelet-config\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:18.981030 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979705 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bns9g\" (UniqueName: \"kubernetes.io/projected/1c471f71-4a76-4003-9f66-663b894892c0-kube-api-access-bns9g\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:18.981030 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979758 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2gzd\" (UniqueName: \"kubernetes.io/projected/e70c8827-8921-4231-8144-395f162320c2-kube-api-access-n2gzd\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:18.981030 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979791 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-k8s-cni-cncf-io\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:18.981030 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979843 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-kubernetes\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:18.981030 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979897 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:18.981030 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:18.979932 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-socket-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.003818 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.003793 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 03:57:19.003975 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.003858 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 03:57:19.004030 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.003986 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 03:57:19.004071 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004034 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 03:57:19.004116 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004068 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h7p9v\"" Oct 30 03:57:19.004163 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004139 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 03:57:19.004285 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004270 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 03:57:19.004362 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004348 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 03:57:19.004413 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004383 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w4blj\"" Oct 30 03:57:19.004413 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004271 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 03:57:19.004531 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004475 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 03:57:19.004531 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004508 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 03:57:19.004595 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004550 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 03:57:19.004595 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004562 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 03:57:19.004654 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004598 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 03:57:19.004654 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004614 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vm8lh\"" Oct 30 03:57:19.004654 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004619 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:57:19.004654 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004644 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 03:57:19.004807 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004681 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 03:57:19.004807 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.004691 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4xznk\"" Oct 30 03:57:19.016565 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.016538 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 03:57:19.016968 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.016925 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kwf82\"" Oct 30 03:57:19.017113 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.016975 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 03:57:19.017113 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.017057 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 03:57:19.017113 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.017075 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 03:57:19.017332 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.017126 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rs54w\"" Oct 30 03:57:19.017332 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.017126 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 03:57:19.017332 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.017142 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 03:57:19.017332 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.017174 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 03:57:19.017332 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.017205 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-smvt7\"" Oct 30 03:57:19.080705 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080673 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93cbeaeb-857f-4c3f-825f-62c46320edd0-kubelet-config\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:19.080858 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080712 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bns9g\" (UniqueName: \"kubernetes.io/projected/1c471f71-4a76-4003-9f66-663b894892c0-kube-api-access-bns9g\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.080858 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080732 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2gzd\" (UniqueName: \"kubernetes.io/projected/e70c8827-8921-4231-8144-395f162320c2-kube-api-access-n2gzd\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:19.080858 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080753 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-k8s-cni-cncf-io\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.080858 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080771 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-kubernetes\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.080858 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080770 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93cbeaeb-857f-4c3f-825f-62c46320edd0-kubelet-config\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:19.080858 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.080858 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080819 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-socket-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.080858 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080836 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-k8s-cni-cncf-io\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.080858 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080842 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-system-cni-dir\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080864 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-kubelet\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080889 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-run-netns\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080893 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080914 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-ovn\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080942 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-ovnkube-script-lib\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080947 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-kubernetes\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62ebb448-14fe-4919-9686-018754ede9cf-ovn-node-metrics-cert\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080994 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-modprobe-d\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081017 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-system-cni-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081041 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-netns\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-kubelet\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081071 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-daemon-config\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081092 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-multus-certs\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081119 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081135 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73ae3560-963a-46a6-a052-21db48a98360-host\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081137 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-kubelet\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081149 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-etc-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.081290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081164 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081177 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-tuned\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081193 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081209 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrkf\" (UniqueName: \"kubernetes.io/projected/471b293f-bc2d-4a9e-bddb-aea255ac235e-kube-api-access-twrkf\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081225 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbqs\" (UniqueName: \"kubernetes.io/projected/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-kube-api-access-jnbqs\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081241 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sj9m\" (UniqueName: \"kubernetes.io/projected/91502bdf-1f01-469f-ac9f-64aa13e5563a-kube-api-access-7sj9m\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081266 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysctl-d\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081286 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-sys\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081301 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-var-lib-kubelet\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081325 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd9d674b-c3d9-4e78-bc55-81cedfc32881-konnectivity-ca\") pod \"konnectivity-agent-9gghj\" (UID: \"cd9d674b-c3d9-4e78-bc55-81cedfc32881\") " pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081341 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-hostroot\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081343 2570 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081355 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-systemd-units\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081369 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-log-socket\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081395 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwzs8\" (UniqueName: \"kubernetes.io/projected/62ebb448-14fe-4919-9686-018754ede9cf-kube-api-access-jwzs8\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081432 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-systemd\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081447 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-cnibin\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.082167 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081462 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-slash\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081484 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081508 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-env-overrides\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081532 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrmh\" (UniqueName: \"kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh\") pod \"network-check-target-v5zp7\" (UID: \"438677a3-904a-474e-bbed-4032260a3a3c\") " pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4m9\" (UniqueName: \"kubernetes.io/projected/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-kube-api-access-dc4m9\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081583 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-device-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081634 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-hosts-file\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081645 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysctl-d\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081657 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-cnibin\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081696 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-daemon-config\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081709 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-cnibin\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081733 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-cni-bin\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081760 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-var-lib-kubelet\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081760 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-ovn\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.080994 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-run-netns\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081092 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-socket-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081827 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73ae3560-963a-46a6-a052-21db48a98360-host\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:19.082956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081834 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-log-socket\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081832 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-slash\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081867 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-systemd-units\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081908 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-etc-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081919 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-modprobe-d\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081955 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082205 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-systemd\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082258 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-cnibin\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082283 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd9d674b-c3d9-4e78-bc55-81cedfc32881-konnectivity-ca\") pod \"konnectivity-agent-9gghj\" (UID: \"cd9d674b-c3d9-4e78-bc55-81cedfc32881\") " pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082297 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081697 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-cni-bin\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082336 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-systemd\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082348 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-sys\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082356 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-node-log\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082376 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-run\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082395 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-lib-modules\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082398 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082406 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-ovnkube-script-lib\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.083776 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082455 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysctl-conf\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082482 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-host\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082492 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-system-cni-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082497 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-lib-modules\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082504 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93cbeaeb-857f-4c3f-825f-62c46320edd0-dbus\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082530 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-registration-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082538 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-hostroot\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082541 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-kubelet\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082558 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-sys-fs\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082565 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-node-log\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082584 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-etc-kubernetes\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082609 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjc9\" (UniqueName: \"kubernetes.io/projected/73ae3560-963a-46a6-a052-21db48a98360-kube-api-access-bnjc9\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082633 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-tmp\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082640 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-run\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082656 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082667 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysctl-conf\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082679 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-tmp-dir\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082703 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-host-slash\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:19.084622 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082728 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-conf-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082754 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-cni-bin\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082777 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysconfig\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082802 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97vg\" (UniqueName: \"kubernetes.io/projected/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-kube-api-access-d97vg\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082826 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082851 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082876 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-socket-dir-parent\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082900 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-cni-multus\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082925 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73ae3560-963a-46a6-a052-21db48a98360-serviceca\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082950 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd9d674b-c3d9-4e78-bc55-81cedfc32881-agent-certs\") pod \"konnectivity-agent-9gghj\" (UID: \"cd9d674b-c3d9-4e78-bc55-81cedfc32881\") " pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082967 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082983 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-var-lib-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.082997 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-cni-netd\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083017 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-env-overrides\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083024 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083049 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-cni-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083074 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91502bdf-1f01-469f-ac9f-64aa13e5563a-cni-binary-copy\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.085269 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083098 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-ovnkube-config\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083123 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-iptables-alerter-script\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083241 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083400 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-device-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083448 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-conf-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.083515 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083538 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-host\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083608 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-socket-dir-parent\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.083623 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret podName:93cbeaeb-857f-4c3f-825f-62c46320edd0 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:19.583592243 +0000 UTC m=+3.096255901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret") pod "global-pull-secret-syncer-p2mmh" (UID: "93cbeaeb-857f-4c3f-825f-62c46320edd0") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083626 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-system-cni-dir\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083686 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-cni-multus\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083729 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93cbeaeb-857f-4c3f-825f-62c46320edd0-dbus\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083736 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysconfig\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.081774 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-multus-certs\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083774 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-cni-bin\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083791 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-netns\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083831 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-sys-fs\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.085930 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083830 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-registration-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083875 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-systemd\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083878 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-etc-kubernetes\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083951 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.084006 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-var-lib-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.084026 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-host-slash\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.083538 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-hosts-file\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.084156 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-tmp-dir\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.084378 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-iptables-alerter-script\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.084511 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.084672 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.084675 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-cni-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.084714 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-cni-netd\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.084794 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.084836 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs podName:e70c8827-8921-4231-8144-395f162320c2 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:19.584820468 +0000 UTC m=+3.097484141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs") pod "network-metrics-daemon-tffth" (UID: "e70c8827-8921-4231-8144-395f162320c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.085302 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.086519 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.085461 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73ae3560-963a-46a6-a052-21db48a98360-serviceca\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:19.087195 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.086564 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-ovnkube-config\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.087195 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.086575 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91502bdf-1f01-469f-ac9f-64aa13e5563a-cni-binary-copy\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.087195 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.086564 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62ebb448-14fe-4919-9686-018754ede9cf-ovn-node-metrics-cert\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.087195 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.086841 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-tuned\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.087195 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.086995 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-tmp\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.087499 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.087478 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd9d674b-c3d9-4e78-bc55-81cedfc32881-agent-certs\") pod \"konnectivity-agent-9gghj\" (UID: \"cd9d674b-c3d9-4e78-bc55-81cedfc32881\") " pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:19.105917 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.105865 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:57:19.105917 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.105893 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:57:19.105917 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.105907 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4lrmh for pod openshift-network-diagnostics/network-check-target-v5zp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:57:19.106110 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.105960 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh podName:438677a3-904a-474e-bbed-4032260a3a3c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:19.605944742 +0000 UTC m=+3.118608398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4lrmh" (UniqueName: "kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh") pod "network-check-target-v5zp7" (UID: "438677a3-904a-474e-bbed-4032260a3a3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:57:19.108342 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.108314 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2gzd\" (UniqueName: \"kubernetes.io/projected/e70c8827-8921-4231-8144-395f162320c2-kube-api-access-n2gzd\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:19.108468 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.108444 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bns9g\" (UniqueName: \"kubernetes.io/projected/1c471f71-4a76-4003-9f66-663b894892c0-kube-api-access-bns9g\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.108562 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.108542 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnbqs\" (UniqueName: \"kubernetes.io/projected/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-kube-api-access-jnbqs\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:19.108770 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.108751 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4m9\" (UniqueName: \"kubernetes.io/projected/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-kube-api-access-dc4m9\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:19.108829 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.108786 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twrkf\" (UniqueName: \"kubernetes.io/projected/471b293f-bc2d-4a9e-bddb-aea255ac235e-kube-api-access-twrkf\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.108891 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.108874 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjc9\" (UniqueName: \"kubernetes.io/projected/73ae3560-963a-46a6-a052-21db48a98360-kube-api-access-bnjc9\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:19.109169 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.109155 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sj9m\" (UniqueName: \"kubernetes.io/projected/91502bdf-1f01-469f-ac9f-64aa13e5563a-kube-api-access-7sj9m\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.113003 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.112984 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwzs8\" (UniqueName: \"kubernetes.io/projected/62ebb448-14fe-4919-9686-018754ede9cf-kube-api-access-jwzs8\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.113079 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.113040 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97vg\" (UniqueName: \"kubernetes.io/projected/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-kube-api-access-d97vg\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.135361 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.135343 2570 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:57:19.199651 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.199625 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:19.206892 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.206870 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:19.210511 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:19.210490 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9baf7ee6_a95f_42bc_8ddf_a197fb3737ff.slice/crio-7652c8208c79486316345b9dac61780568c4365251e86fdf87a01dacfdf1dedf WatchSource:0}: Error finding container 7652c8208c79486316345b9dac61780568c4365251e86fdf87a01dacfdf1dedf: Status 404 returned error can't find the container with id 7652c8208c79486316345b9dac61780568c4365251e86fdf87a01dacfdf1dedf Oct 30 03:57:19.218229 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:19.218205 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9d674b_c3d9_4e78_bc55_81cedfc32881.slice/crio-aee539f33e518dc52781f4c17bd7d1a48147663fcd223b756f32a58cf8a17e7c WatchSource:0}: Error finding container aee539f33e518dc52781f4c17bd7d1a48147663fcd223b756f32a58cf8a17e7c: Status 404 returned error can't find the container with id aee539f33e518dc52781f4c17bd7d1a48147663fcd223b756f32a58cf8a17e7c Oct 30 03:57:19.224670 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.224637 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:19.232743 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.232712 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:19.234104 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:19.234072 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c3b257_dd0c_4af8_aa03_3efb4fb6a384.slice/crio-62b0c2bb8e22a9ee06b1009118d960ea8359a4b723d4e8a349590afee949810d WatchSource:0}: Error finding container 62b0c2bb8e22a9ee06b1009118d960ea8359a4b723d4e8a349590afee949810d: Status 404 returned error can't find the container with id 62b0c2bb8e22a9ee06b1009118d960ea8359a4b723d4e8a349590afee949810d Oct 30 03:57:19.256313 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.256292 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:19.259427 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.259390 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:19.265762 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.265734 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9fmv2" Oct 30 03:57:19.266989 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.266971 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:19.272132 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:19.271737 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db45ae1_4ba3_466f_82f9_005ce3d82bd2.slice/crio-4943309beae3fa168b16fa6651ee168faea5dabc32e9fa322e8de13241bd72aa WatchSource:0}: Error finding container 4943309beae3fa168b16fa6651ee168faea5dabc32e9fa322e8de13241bd72aa: Status 404 returned error can't find the container with id 4943309beae3fa168b16fa6651ee168faea5dabc32e9fa322e8de13241bd72aa Oct 30 03:57:19.273489 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.273466 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:19.277370 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:19.277248 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91502bdf_1f01_469f_ac9f_64aa13e5563a.slice/crio-b323e055211f8138d7a3042ac59e64b439fc2fe9726d4403697c69548f827853 WatchSource:0}: Error finding container b323e055211f8138d7a3042ac59e64b439fc2fe9726d4403697c69548f827853: Status 404 returned error can't find the container with id b323e055211f8138d7a3042ac59e64b439fc2fe9726d4403697c69548f827853 Oct 30 03:57:19.283195 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:19.283168 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ae3560_963a_46a6_a052_21db48a98360.slice/crio-44b8082218dde0674055a08592a46b04f75ca8da879155d82b9f460e35a83f6c WatchSource:0}: Error finding container 44b8082218dde0674055a08592a46b04f75ca8da879155d82b9f460e35a83f6c: Status 404 returned error can't find the container with id 44b8082218dde0674055a08592a46b04f75ca8da879155d82b9f460e35a83f6c Oct 30 03:57:19.286836 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:19.286524 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471b293f_bc2d_4a9e_bddb_aea255ac235e.slice/crio-c1befeb0a3b86351b7c14a4e2e2d45c17eecbd5b0b65949d1d0be4616abcc6fb WatchSource:0}: Error finding container c1befeb0a3b86351b7c14a4e2e2d45c17eecbd5b0b65949d1d0be4616abcc6fb: Status 404 returned error can't find the container with id c1befeb0a3b86351b7c14a4e2e2d45c17eecbd5b0b65949d1d0be4616abcc6fb Oct 30 03:57:19.587247 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.587215 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:19.587581 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.587277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:19.587581 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.587379 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:57:19.587581 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.587396 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:57:19.587581 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.587480 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret podName:93cbeaeb-857f-4c3f-825f-62c46320edd0 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:20.58746142 +0000 UTC m=+4.100125068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret") pod "global-pull-secret-syncer-p2mmh" (UID: "93cbeaeb-857f-4c3f-825f-62c46320edd0") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:57:19.587581 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.587504 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs podName:e70c8827-8921-4231-8144-395f162320c2 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:20.587493436 +0000 UTC m=+4.100157086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs") pod "network-metrics-daemon-tffth" (UID: "e70c8827-8921-4231-8144-395f162320c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:57:19.689221 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.688587 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrmh\" (UniqueName: \"kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh\") pod \"network-check-target-v5zp7\" (UID: \"438677a3-904a-474e-bbed-4032260a3a3c\") " pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:19.689221 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.688788 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:57:19.689221 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.688806 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:57:19.689221 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.688818 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4lrmh for pod openshift-network-diagnostics/network-check-target-v5zp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:57:19.689221 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.688870 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh podName:438677a3-904a-474e-bbed-4032260a3a3c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:20.688853425 +0000 UTC m=+4.201517095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4lrmh" (UniqueName: "kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh") pod "network-check-target-v5zp7" (UID: "438677a3-904a-474e-bbed-4032260a3a3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:57:19.927557 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.927494 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 03:52:18 +0000 UTC" deadline="2028-06-25 01:25:08.321952224 +0000 UTC" Oct 30 03:57:19.927557 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.927554 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14493h27m48.394402471s" Oct 30 03:57:19.935011 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.934987 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:19.935147 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:19.935115 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:19.963553 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.963482 2570 generic.go:356] "Generic (PLEG): container finished" podID="3a1a550d76a223b097bf2829db01e97d" containerID="8dea47e83590faec28e80a182a13f3a67685e6d1a90a928b80c184d4e7171e1a" exitCode=0 Oct 30 03:57:19.963675 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.963570 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" event={"ID":"3a1a550d76a223b097bf2829db01e97d","Type":"ContainerDied","Data":"8dea47e83590faec28e80a182a13f3a67685e6d1a90a928b80c184d4e7171e1a"} Oct 30 03:57:19.974561 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.974533 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x58t9" event={"ID":"73ae3560-963a-46a6-a052-21db48a98360","Type":"ContainerStarted","Data":"44b8082218dde0674055a08592a46b04f75ca8da879155d82b9f460e35a83f6c"} Oct 30 03:57:19.986723 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:19.986651 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zjzzn" event={"ID":"6db45ae1-4ba3-466f-82f9-005ce3d82bd2","Type":"ContainerStarted","Data":"4943309beae3fa168b16fa6651ee168faea5dabc32e9fa322e8de13241bd72aa"} Oct 30 03:57:20.007183 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.007097 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9gghj" event={"ID":"cd9d674b-c3d9-4e78-bc55-81cedfc32881","Type":"ContainerStarted","Data":"aee539f33e518dc52781f4c17bd7d1a48147663fcd223b756f32a58cf8a17e7c"} Oct 30 03:57:20.026186 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.026155 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpm6b" event={"ID":"471b293f-bc2d-4a9e-bddb-aea255ac235e","Type":"ContainerStarted","Data":"c1befeb0a3b86351b7c14a4e2e2d45c17eecbd5b0b65949d1d0be4616abcc6fb"} Oct 30 03:57:20.032552 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.032527 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9fmv2" event={"ID":"91502bdf-1f01-469f-ac9f-64aa13e5563a","Type":"ContainerStarted","Data":"b323e055211f8138d7a3042ac59e64b439fc2fe9726d4403697c69548f827853"} Oct 30 03:57:20.045961 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.045873 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" event={"ID":"62ebb448-14fe-4919-9686-018754ede9cf","Type":"ContainerStarted","Data":"004d1a87951da880edfce6abcf22b6583af05d032b6b9e1af79ea47b03f5fc85"} Oct 30 03:57:20.057656 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.057623 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" event={"ID":"1c471f71-4a76-4003-9f66-663b894892c0","Type":"ContainerStarted","Data":"f9e3a029cd0b30c569bc987d45c211a622d84cbe6a04cf31e8ec796f51f14273"} Oct 30 03:57:20.068958 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.068934 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q98qf" event={"ID":"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384","Type":"ContainerStarted","Data":"62b0c2bb8e22a9ee06b1009118d960ea8359a4b723d4e8a349590afee949810d"} Oct 30 03:57:20.085575 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.085547 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" event={"ID":"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff","Type":"ContainerStarted","Data":"7652c8208c79486316345b9dac61780568c4365251e86fdf87a01dacfdf1dedf"} Oct 30 03:57:20.121756 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.121729 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" event={"ID":"552cd6bdf9ef61710ec1ae6f1eae6b32","Type":"ContainerStarted","Data":"1fe70b0f577fc3d5cb4af58dc38cbee39a911c41090b483da763b153fd5ee3d9"} Oct 30 03:57:20.161211 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.161156 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" podStartSLOduration=2.161141197 podStartE2EDuration="2.161141197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:20.160623796 +0000 UTC m=+3.673287466" watchObservedRunningTime="2026-10-30 03:57:20.161141197 +0000 UTC m=+3.673804865" Oct 30 03:57:20.615561 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.615482 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:20.615561 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.615557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:20.615766 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:20.615683 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:57:20.615766 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:20.615739 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs podName:e70c8827-8921-4231-8144-395f162320c2 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:22.615721952 +0000 UTC m=+6.128385611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs") pod "network-metrics-daemon-tffth" (UID: "e70c8827-8921-4231-8144-395f162320c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:57:20.616132 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:20.616113 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:57:20.616208 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:20.616166 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret podName:93cbeaeb-857f-4c3f-825f-62c46320edd0 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:22.616150775 +0000 UTC m=+6.128814445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret") pod "global-pull-secret-syncer-p2mmh" (UID: "93cbeaeb-857f-4c3f-825f-62c46320edd0") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:57:20.715893 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.715857 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrmh\" (UniqueName: \"kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh\") pod \"network-check-target-v5zp7\" (UID: \"438677a3-904a-474e-bbed-4032260a3a3c\") " pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:20.716097 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:20.716080 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:57:20.716160 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:20.716106 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:57:20.716160 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:20.716119 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4lrmh for pod openshift-network-diagnostics/network-check-target-v5zp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:57:20.716263 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:20.716170 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh podName:438677a3-904a-474e-bbed-4032260a3a3c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:22.716154695 +0000 UTC m=+6.228818345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4lrmh" (UniqueName: "kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh") pod "network-check-target-v5zp7" (UID: "438677a3-904a-474e-bbed-4032260a3a3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:57:20.927946 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.927835 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 03:52:18 +0000 UTC" deadline="2028-06-06 22:33:48.678837274 +0000 UTC" Oct 30 03:57:20.927946 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.927899 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14058h36m27.750942246s" Oct 30 03:57:20.936659 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.936636 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:20.936798 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:20.936753 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:20.937152 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:20.937131 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:20.937269 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:20.937244 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:21.134435 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:21.134384 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" event={"ID":"3a1a550d76a223b097bf2829db01e97d","Type":"ContainerStarted","Data":"0e9b6b5d462ce55533fa5ebc9d065585ad252f3f87a3ff6a1fc81716610bad79"} Oct 30 03:57:21.179012 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:21.178914 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" podStartSLOduration=3.178901276 podStartE2EDuration="3.178901276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:21.178190431 +0000 UTC m=+4.690854098" watchObservedRunningTime="2026-10-30 03:57:21.178901276 +0000 UTC m=+4.691564941" Oct 30 03:57:21.935339 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:21.934891 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:21.935339 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:21.935003 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:22.142020 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.141194 2570 generic.go:356] "Generic (PLEG): container finished" podID="471b293f-bc2d-4a9e-bddb-aea255ac235e" containerID="219c8215555d34603f9393a89a39e4019dc6ca2fe649c10a6a9dc28f3ae2bec2" exitCode=0 Oct 30 03:57:22.142020 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.141246 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpm6b" event={"ID":"471b293f-bc2d-4a9e-bddb-aea255ac235e","Type":"ContainerDied","Data":"219c8215555d34603f9393a89a39e4019dc6ca2fe649c10a6a9dc28f3ae2bec2"} Oct 30 03:57:22.630997 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.630924 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:22.630997 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.630990 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:22.631203 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:22.631135 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:57:22.631251 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:22.631204 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs podName:e70c8827-8921-4231-8144-395f162320c2 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:26.631176678 +0000 UTC m=+10.143840336 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs") pod "network-metrics-daemon-tffth" (UID: "e70c8827-8921-4231-8144-395f162320c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:57:22.631653 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:22.631628 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:57:22.631766 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:22.631690 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret podName:93cbeaeb-857f-4c3f-825f-62c46320edd0 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:26.631672784 +0000 UTC m=+10.144336450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret") pod "global-pull-secret-syncer-p2mmh" (UID: "93cbeaeb-857f-4c3f-825f-62c46320edd0") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:57:22.732339 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.732303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrmh\" (UniqueName: \"kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh\") pod \"network-check-target-v5zp7\" (UID: \"438677a3-904a-474e-bbed-4032260a3a3c\") " pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:22.732555 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:22.732537 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:57:22.732653 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:22.732563 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:57:22.732653 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:22.732578 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4lrmh for pod openshift-network-diagnostics/network-check-target-v5zp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:57:22.732653 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:22.732632 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh podName:438677a3-904a-474e-bbed-4032260a3a3c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:26.732614525 +0000 UTC m=+10.245278189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4lrmh" (UniqueName: "kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh") pod "network-check-target-v5zp7" (UID: "438677a3-904a-474e-bbed-4032260a3a3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:57:22.815546 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.815489 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d8rtc"] Oct 30 03:57:22.822083 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.822063 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:22.857685 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.857654 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:57:22.857864 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.857848 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 03:57:22.857961 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.857939 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:57:22.858039 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.858024 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 03:57:22.888781 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.888732 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 03:57:22.888907 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.888817 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 03:57:22.888907 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.888889 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8s8mj\"" Oct 30 03:57:22.937065 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.937031 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:22.937428 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:22.937159 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:22.937572 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.937534 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:22.937650 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:22.937628 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:22.937701 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.937664 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-tls\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:22.937750 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.937738 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-root\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:22.937796 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.937763 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-textfile\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:22.937855 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.937792 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2b5z\" (UniqueName: \"kubernetes.io/projected/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-kube-api-access-t2b5z\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:22.937855 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.937820 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:22.937951 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.937874 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-sys\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:22.937951 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.937936 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-wtmp\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:22.938044 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.937957 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:22.938095 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:22.938032 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-metrics-client-ca\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.038690 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.038659 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-root\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.038829 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.038700 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-textfile\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.038829 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.038726 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2b5z\" (UniqueName: \"kubernetes.io/projected/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-kube-api-access-t2b5z\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.038829 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.038749 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.038829 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.038774 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-sys\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.039042 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.038866 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-wtmp\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.039042 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.038896 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.039042 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.038927 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-metrics-client-ca\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.039042 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.038975 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-tls\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.040106 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.040059 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-sys\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.040106 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.040059 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-root\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.040267 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.040177 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-textfile\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.040548 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.040528 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-wtmp\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.040640 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.040578 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-metrics-client-ca\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.040640 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.040599 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.043373 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.043352 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-tls\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.045820 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.045798 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.072672 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.072646 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2b5z\" (UniqueName: \"kubernetes.io/projected/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-kube-api-access-t2b5z\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.144389 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.144317 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:23.148882 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.148562 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9gghj" event={"ID":"cd9d674b-c3d9-4e78-bc55-81cedfc32881","Type":"ContainerStarted","Data":"bc689e61de8872f7367e26280be631c86b38e214442da49ceb5c1ce5c1f0c474"} Oct 30 03:57:23.181653 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:23.181626 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703f4255_11e5_4a34_a7d2_c1e26ef1fe3d.slice/crio-ae5cb5e8f5b97a78add2edabad7c2e76ae5fa5c3fe25ef9df4910ca1bd2045d3 WatchSource:0}: Error finding container ae5cb5e8f5b97a78add2edabad7c2e76ae5fa5c3fe25ef9df4910ca1bd2045d3: Status 404 returned error can't find the container with id ae5cb5e8f5b97a78add2edabad7c2e76ae5fa5c3fe25ef9df4910ca1bd2045d3 Oct 30 03:57:23.350043 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.349929 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:23.574073 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.573981 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:23.935475 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.934997 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:23.935475 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:23.935116 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:23.975503 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:23.975476 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:24.157191 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:24.157161 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8rtc" event={"ID":"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d","Type":"ContainerStarted","Data":"ae5cb5e8f5b97a78add2edabad7c2e76ae5fa5c3fe25ef9df4910ca1bd2045d3"} Oct 30 03:57:24.202037 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:24.201949 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:24.422820 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:24.422706 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:24.619515 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:24.619375 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:24.620349 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:24.620321 2570 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:24.671466 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:24.671060 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9gghj" podStartSLOduration=4.234090276 podStartE2EDuration="7.671040529s" totalImagesPullingTime="3.436950253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:23.191619881 +0000 UTC m=+6.704283550" watchObservedRunningTime="2026-10-30 03:57:24.671040529 +0000 UTC m=+8.183704194" Oct 30 03:57:24.679451 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:24.678686 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:24.936697 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:24.936671 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:24.936821 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:24.936799 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:24.936884 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:24.936856 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:24.936941 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:24.936924 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:25.161537 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:25.161365 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" event={"ID":"1c471f71-4a76-4003-9f66-663b894892c0","Type":"ContainerStarted","Data":"c544dbc90e17b0a54bca6c5366ed4de8e416a4f487ce8fc82acd967b3cdfeec9"} Oct 30 03:57:25.164528 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:25.164495 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x58t9" event={"ID":"73ae3560-963a-46a6-a052-21db48a98360","Type":"ContainerStarted","Data":"839b0c156c72b70bd0db0962f1b53136ad3b1286045580b66ddc5d989c3dc480"} Oct 30 03:57:25.934794 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:25.934763 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:25.934956 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:25.934873 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:26.135151 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.135113 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:26.173441 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.171735 2570 generic.go:356] "Generic (PLEG): container finished" podID="703f4255-11e5-4a34-a7d2-c1e26ef1fe3d" containerID="c6997d00e1f3e286e39cdf6b42c1af4c760dd15811b86c4deb6338dc2236ebfd" exitCode=0 Oct 30 03:57:26.173441 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.171837 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 03:57:26.173441 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.172817 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8rtc" event={"ID":"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d","Type":"ContainerDied","Data":"c6997d00e1f3e286e39cdf6b42c1af4c760dd15811b86c4deb6338dc2236ebfd"} Oct 30 03:57:26.230104 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.230050 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x58t9" podStartSLOduration=4.0465216 podStartE2EDuration="9.230030872s" totalImagesPullingTime="5.183509272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:25.204752783 +0000 UTC m=+8.717416451" watchObservedRunningTime="2026-10-30 03:57:26.230030872 +0000 UTC m=+9.742694534" Oct 30 03:57:26.355949 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.355916 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:26.426709 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.426619 2570 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 03:57:26.584883 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.584688 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:26.671659 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.671620 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:26.671829 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.671685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:26.671829 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:26.671780 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:57:26.671938 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:26.671833 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret podName:93cbeaeb-857f-4c3f-825f-62c46320edd0 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:34.671818672 +0000 UTC m=+18.184482316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret") pod "global-pull-secret-syncer-p2mmh" (UID: "93cbeaeb-857f-4c3f-825f-62c46320edd0") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:57:26.672173 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:26.672152 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:57:26.672271 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:26.672212 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs podName:e70c8827-8921-4231-8144-395f162320c2 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:34.672195281 +0000 UTC m=+18.184858930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs") pod "network-metrics-daemon-tffth" (UID: "e70c8827-8921-4231-8144-395f162320c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:57:26.772090 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.772055 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrmh\" (UniqueName: \"kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh\") pod \"network-check-target-v5zp7\" (UID: \"438677a3-904a-474e-bbed-4032260a3a3c\") " pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:26.772284 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:26.772264 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:57:26.772350 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:26.772293 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:57:26.772350 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:26.772308 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4lrmh for pod openshift-network-diagnostics/network-check-target-v5zp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:57:26.772471 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:26.772367 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh podName:438677a3-904a-474e-bbed-4032260a3a3c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:34.772348613 +0000 UTC m=+18.285012259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4lrmh" (UniqueName: "kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh") pod "network-check-target-v5zp7" (UID: "438677a3-904a-474e-bbed-4032260a3a3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:57:26.809401 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.809357 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:26.937109 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.937039 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:26.937263 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.937214 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:26.937361 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:26.937336 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:26.937446 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:26.937210 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:26.938647 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.938540 2570 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T03:57:26.426636972Z","UUID":"767f9a44-928d-4e46-a3d6-a0a8753d67c4","Handler":null,"Name":"","Endpoint":""} Oct 30 03:57:26.941852 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.941808 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 03:57:26.941852 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:26.941835 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 03:57:27.030565 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:27.030323 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:27.183898 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:27.183864 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8rtc" event={"ID":"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d","Type":"ContainerStarted","Data":"08b1d421e55c9f9037ae74f1d25cf808292c30b44d0334dfccc95d34c04d63d8"} Oct 30 03:57:27.184305 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:27.183935 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d8rtc" event={"ID":"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d","Type":"ContainerStarted","Data":"1974252281cd24394aa6103be4d8c92ba0ff55407ebb010d2a7aefe2be1764e8"} Oct 30 03:57:27.186811 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:27.186781 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9fmv2" event={"ID":"91502bdf-1f01-469f-ac9f-64aa13e5563a","Type":"ContainerStarted","Data":"f7f2e050e85727d0ca0be213b4b40e8fa72c8fe94fe132a922ff1b5681e4244b"} Oct 30 03:57:27.193099 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:27.193004 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" event={"ID":"1c471f71-4a76-4003-9f66-663b894892c0","Type":"ContainerStarted","Data":"359ef3cd12c3836dd23f817404b6f7293c46544007630da6f45e496f1f886126"} Oct 30 03:57:27.234739 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:27.234688 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d8rtc" podStartSLOduration=3.113671767 podStartE2EDuration="5.23467311s" totalImagesPullingTime="2.028724892s" totalInitContainerRuntime="92.276451ms" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:27.234597228 +0000 UTC m=+10.747260894" watchObservedRunningTime="2026-10-30 03:57:27.23467311 +0000 UTC m=+10.747336779" Oct 30 03:57:27.502760 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:27.502726 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:27.712543 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:27.712512 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:27.935054 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:27.935028 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:27.935191 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:27.935125 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:27.956699 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:27.956671 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:28.198648 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:28.198571 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" event={"ID":"1c471f71-4a76-4003-9f66-663b894892c0","Type":"ContainerStarted","Data":"9129c43ded455bb95d7bae95aaad8efcee2a2edbfb64d514a8d0462dfdd694c4"} Oct 30 03:57:28.200463 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:28.200408 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zjzzn" event={"ID":"6db45ae1-4ba3-466f-82f9-005ce3d82bd2","Type":"ContainerStarted","Data":"f799ae3a8cb10e789104338f14968196bbadd62241803c150971fe4e1dd86dfe"} Oct 30 03:57:28.202762 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:28.202736 2570 generic.go:356] "Generic (PLEG): container finished" podID="471b293f-bc2d-4a9e-bddb-aea255ac235e" containerID="0bc5890876fabbc48ce7d8ebb357116880d35c0d394917910289d79a9bba3883" exitCode=0 Oct 30 03:57:28.203026 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:28.202997 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpm6b" event={"ID":"471b293f-bc2d-4a9e-bddb-aea255ac235e","Type":"ContainerDied","Data":"0bc5890876fabbc48ce7d8ebb357116880d35c0d394917910289d79a9bba3883"} Oct 30 03:57:28.241687 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:28.241631 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9fmv2" podStartSLOduration=3.574896951 podStartE2EDuration="11.241612922s" totalImagesPullingTime="7.666715971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:27.280150017 +0000 UTC m=+10.792813686" watchObservedRunningTime="2026-10-30 03:57:28.241612922 +0000 UTC m=+11.754276608" Oct 30 03:57:28.242397 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:28.242363 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" podStartSLOduration=3.27775833 podStartE2EDuration="11.242356124s" totalImagesPullingTime="7.964597794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:28.24136117 +0000 UTC m=+11.754024838" watchObservedRunningTime="2026-10-30 03:57:28.242356124 +0000 UTC m=+11.755019790" Oct 30 03:57:28.281588 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:28.281540 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zjzzn" podStartSLOduration=3.30501398 podStartE2EDuration="11.281525009s" totalImagesPullingTime="7.976511029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:28.280605093 +0000 UTC m=+11.793268760" watchObservedRunningTime="2026-10-30 03:57:28.281525009 +0000 UTC m=+11.794188677" Oct 30 03:57:28.882317 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:28.882244 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:28.935181 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:28.935144 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:28.935181 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:28.935192 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:28.935471 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:28.935447 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:28.935770 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:28.935309 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:29.102239 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:29.102211 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:29.206990 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:29.206962 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q98qf" event={"ID":"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384","Type":"ContainerStarted","Data":"973bd0a3c60fcf350563922e85bf20e6c584f14f0256fdb87f90e32f433a1cba"} Oct 30 03:57:29.247897 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:29.247853 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q98qf" podStartSLOduration=4.385413322 podStartE2EDuration="12.247839033s" totalImagesPullingTime="7.862425711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:29.247010607 +0000 UTC m=+12.759674273" watchObservedRunningTime="2026-10-30 03:57:29.247839033 +0000 UTC m=+12.760502699" Oct 30 03:57:29.439203 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:29.439130 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:29.662708 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:29.662489 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:29.934901 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:29.934800 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:29.935051 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:29.934912 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:29.941037 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:29.941007 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:30.211549 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:30.211474 2570 generic.go:356] "Generic (PLEG): container finished" podID="471b293f-bc2d-4a9e-bddb-aea255ac235e" containerID="dbe1b27ab3dc1c8f833f3ee9600a1460a3a61d7937b175d4b2e9e4bb9b2faa2d" exitCode=0 Oct 30 03:57:30.212177 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:30.211552 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpm6b" event={"ID":"471b293f-bc2d-4a9e-bddb-aea255ac235e","Type":"ContainerDied","Data":"dbe1b27ab3dc1c8f833f3ee9600a1460a3a61d7937b175d4b2e9e4bb9b2faa2d"} Oct 30 03:57:30.213319 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:30.213297 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" event={"ID":"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff","Type":"ContainerStarted","Data":"1dcff88b854652d412782d0edc934cec0dd5db6a61c1be62787c893012f148c9"} Oct 30 03:57:30.326179 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:30.326126 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" podStartSLOduration=3.069436017 podStartE2EDuration="13.326112758s" totalImagesPullingTime="10.256676741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:30.325463106 +0000 UTC m=+13.838126772" watchObservedRunningTime="2026-10-30 03:57:30.326112758 +0000 UTC m=+13.838776424" Oct 30 03:57:30.934769 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:30.934745 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:30.934897 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:30.934877 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:30.935308 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:30.935288 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:30.935436 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:30.935394 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:31.218065 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:31.217981 2570 generic.go:356] "Generic (PLEG): container finished" podID="471b293f-bc2d-4a9e-bddb-aea255ac235e" containerID="65aad0390297bd80d1f1d3e14abeffc0ec823691dc4543d0ed79cb93aa22f42e" exitCode=0 Oct 30 03:57:31.218458 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:31.218070 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpm6b" event={"ID":"471b293f-bc2d-4a9e-bddb-aea255ac235e","Type":"ContainerDied","Data":"65aad0390297bd80d1f1d3e14abeffc0ec823691dc4543d0ed79cb93aa22f42e"} Oct 30 03:57:31.934694 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:31.934662 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:31.934865 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:31.934796 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:32.934178 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:32.934151 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:32.934608 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:32.934189 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:32.934608 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:32.934281 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:32.934608 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:32.934379 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:33.107348 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:33.107278 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:33.224985 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:33.224953 2570 generic.go:356] "Generic (PLEG): container finished" podID="471b293f-bc2d-4a9e-bddb-aea255ac235e" containerID="29483355a2f69cb0135447c5a6436782f670670e09fc35df11bd2a71548d40c4" exitCode=0 Oct 30 03:57:33.225095 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:33.224993 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpm6b" event={"ID":"471b293f-bc2d-4a9e-bddb-aea255ac235e","Type":"ContainerDied","Data":"29483355a2f69cb0135447c5a6436782f670670e09fc35df11bd2a71548d40c4"} Oct 30 03:57:33.408155 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:33.408131 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:33.934717 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:33.934562 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:33.935122 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:33.934812 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:34.118120 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.118098 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:34.118240 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.118201 2570 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 03:57:34.118666 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.118641 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:34.230276 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.230252 2570 generic.go:356] "Generic (PLEG): container finished" podID="471b293f-bc2d-4a9e-bddb-aea255ac235e" containerID="6cb6adf2935a3db4a727d643af264bbbacbf85516251e5b8a4248b9ef6cb5ada" exitCode=0 Oct 30 03:57:34.230403 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.230326 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpm6b" event={"ID":"471b293f-bc2d-4a9e-bddb-aea255ac235e","Type":"ContainerDied","Data":"6cb6adf2935a3db4a727d643af264bbbacbf85516251e5b8a4248b9ef6cb5ada"} Oct 30 03:57:34.233325 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.233299 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" event={"ID":"62ebb448-14fe-4919-9686-018754ede9cf","Type":"ContainerStarted","Data":"04ebbe3c6af8e6d60cb94f1bfd1cbf09b8f441ed9d688d5391d0fecd7b4f0bf2"} Oct 30 03:57:34.233400 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.233364 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" event={"ID":"62ebb448-14fe-4919-9686-018754ede9cf","Type":"ContainerStarted","Data":"9f7a79df7f2dfca84c127df388564ed06e52b89ecb9340ab9a87b446360bb07d"} Oct 30 03:57:34.233400 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.233395 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" event={"ID":"62ebb448-14fe-4919-9686-018754ede9cf","Type":"ContainerStarted","Data":"c4423dcc877edf7ca42df209ab10cb240ce9eca2750fd2c64189a5824fac0468"} Oct 30 03:57:34.233478 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.233444 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" event={"ID":"62ebb448-14fe-4919-9686-018754ede9cf","Type":"ContainerStarted","Data":"3fca7e76351bb818c7c23c7c868fecf6e995a1fa84065aec0a6dea0bea4f664e"} Oct 30 03:57:34.233478 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.233473 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" event={"ID":"62ebb448-14fe-4919-9686-018754ede9cf","Type":"ContainerStarted","Data":"1fd43235c5fccefad875f7dd34d10d813ab4b7f779080912583a4b1c63323940"} Oct 30 03:57:34.233554 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.233500 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" event={"ID":"62ebb448-14fe-4919-9686-018754ede9cf","Type":"ContainerStarted","Data":"102efef444b97365ccdccb9863f7d97e1aa4b61061841573083be9f2fe00f0dd"} Oct 30 03:57:34.403972 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.403947 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:34.731025 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.730954 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:34.731025 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.731002 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:34.731179 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:34.731083 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 03:57:34.731179 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:34.731085 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:57:34.731179 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:34.731126 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret podName:93cbeaeb-857f-4c3f-825f-62c46320edd0 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:50.731114648 +0000 UTC m=+34.243778293 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret") pod "global-pull-secret-syncer-p2mmh" (UID: "93cbeaeb-857f-4c3f-825f-62c46320edd0") : object "kube-system"/"original-pull-secret" not registered Oct 30 03:57:34.731179 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:34.731139 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs podName:e70c8827-8921-4231-8144-395f162320c2 nodeName:}" failed. No retries permitted until 2026-10-30 03:57:50.731132992 +0000 UTC m=+34.243796636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs") pod "network-metrics-daemon-tffth" (UID: "e70c8827-8921-4231-8144-395f162320c2") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 03:57:34.832034 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.831999 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrmh\" (UniqueName: \"kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh\") pod \"network-check-target-v5zp7\" (UID: \"438677a3-904a-474e-bbed-4032260a3a3c\") " pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:34.832187 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:34.832127 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 03:57:34.832187 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:34.832143 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 03:57:34.832187 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:34.832151 2570 projected.go:196] Error preparing data for projected volume kube-api-access-4lrmh for pod openshift-network-diagnostics/network-check-target-v5zp7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:57:34.832280 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:34.832191 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh podName:438677a3-904a-474e-bbed-4032260a3a3c nodeName:}" failed. No retries permitted until 2026-10-30 03:57:50.832179738 +0000 UTC m=+34.344843382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4lrmh" (UniqueName: "kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh") pod "network-check-target-v5zp7" (UID: "438677a3-904a-474e-bbed-4032260a3a3c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 03:57:34.934274 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.934249 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:34.934402 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:34.934358 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:34.934504 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:34.934466 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:34.934574 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:34.934554 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:35.238344 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:35.238316 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lpm6b" event={"ID":"471b293f-bc2d-4a9e-bddb-aea255ac235e","Type":"ContainerStarted","Data":"4847110f5d4378ae039f01f798d0329ad54e9fe96f94b5ca7ce9b87ce830de87"} Oct 30 03:57:35.285682 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:35.285645 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lpm6b" podStartSLOduration=7.758726258 podStartE2EDuration="18.285632437s" totalImagesPullingTime="10.526906179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:35.284883581 +0000 UTC m=+18.797547246" watchObservedRunningTime="2026-10-30 03:57:35.285632437 +0000 UTC m=+18.798296103" Oct 30 03:57:35.701934 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:35.701861 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:35.919546 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:35.919520 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:35.934350 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:35.934331 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:35.934452 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:35.934431 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:36.247124 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:36.247085 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" event={"ID":"62ebb448-14fe-4919-9686-018754ede9cf","Type":"ContainerStarted","Data":"ab2c5a06c7e44d67f555ddfefadd065247d9c93834afadc79f35baede8a67eee"} Oct 30 03:57:36.935823 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:36.935794 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:36.936002 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:36.935983 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:36.936002 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:36.935981 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:36.936106 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:36.936083 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:37.378499 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:37.378386 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:37.600956 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:37.600928 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:37.924811 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:37.924783 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:37.934834 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:37.934816 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:37.934913 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:37.934899 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:38.159255 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:38.159230 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:38.254674 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:38.254631 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" event={"ID":"62ebb448-14fe-4919-9686-018754ede9cf","Type":"ContainerStarted","Data":"70fc281dbd90861e9961dcc8117790fdeff480008f3458264864999c3885018b"} Oct 30 03:57:38.254926 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:38.254902 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:38.255046 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:38.254977 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:38.269440 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:38.269405 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:38.349595 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:38.349545 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" podStartSLOduration=6.881824728 podStartE2EDuration="21.349531529s" totalImagesPullingTime="14.467706801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:38.336365202 +0000 UTC m=+21.849028867" watchObservedRunningTime="2026-10-30 03:57:38.349531529 +0000 UTC m=+21.862195238" Oct 30 03:57:38.934583 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:38.934549 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:38.935097 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:38.934690 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:38.935097 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:38.934724 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:38.935097 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:38.934782 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:39.257132 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:39.257065 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:39.271963 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:39.271942 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:39.934123 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:39.934097 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:39.934267 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:39.934196 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:40.293587 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:40.293529 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p2mmh"] Oct 30 03:57:40.293958 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:40.293611 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:40.293958 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:40.293683 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:40.296770 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:40.296747 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v5zp7"] Oct 30 03:57:40.296887 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:40.296832 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:40.296928 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:40.296906 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:40.297360 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:40.297341 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tffth"] Oct 30 03:57:40.297452 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:40.297440 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:40.297534 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:40.297520 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:41.282313 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:41.282274 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:41.494397 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:41.494370 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:41.934264 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:41.934238 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:41.934414 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:41.934320 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:41.934414 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:41.934343 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:41.934414 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:41.934323 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:41.934568 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:41.934433 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:41.934568 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:41.934484 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:43.902056 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:43.902027 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:43.934248 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:43.934200 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:43.934248 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:43.934243 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:43.934447 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:43.934257 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:43.934447 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:43.934321 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-v5zp7" podUID="438677a3-904a-474e-bbed-4032260a3a3c" Oct 30 03:57:43.934447 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:43.934391 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p2mmh" podUID="93cbeaeb-857f-4c3f-825f-62c46320edd0" Oct 30 03:57:43.934572 ip-10-0-138-177 kubenswrapper[2570]: E1030 03:57:43.934493 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tffth" podUID="e70c8827-8921-4231-8144-395f162320c2" Oct 30 03:57:44.121693 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:44.121625 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:44.343233 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:44.343200 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:45.776482 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.776259 2570 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeReady" Oct 30 03:57:45.776903 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.776544 2570 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 03:57:45.837507 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.837480 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zrv7n"] Oct 30 03:57:45.840804 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.840790 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:45.842951 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.842929 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wq96k"] Oct 30 03:57:45.845569 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.845554 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wq96k" Oct 30 03:57:45.857507 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.857489 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b2qtm\"" Oct 30 03:57:45.857507 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.857492 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 03:57:45.858609 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.858591 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 03:57:45.862656 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.862634 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zrv7n"] Oct 30 03:57:45.865479 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.865461 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wq96k"] Oct 30 03:57:45.934795 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.934771 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:45.934897 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.934884 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:45.935085 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.935065 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:45.938737 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.938718 2570 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b7bml"] Oct 30 03:57:45.943260 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.943243 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:45.957628 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.957608 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 03:57:45.957721 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.957679 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j9mnz\"" Oct 30 03:57:45.957721 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.957681 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vnctk\"" Oct 30 03:57:45.957721 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.957695 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 03:57:45.957876 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.957746 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hmk7j\"" Oct 30 03:57:45.957876 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.957773 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mwxht\"" Oct 30 03:57:45.957974 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.957886 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 03:57:45.958360 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.958342 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 03:57:45.958475 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.958460 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 03:57:45.958534 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.958477 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 03:57:45.958534 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.958477 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 03:57:45.960886 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.960868 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 03:57:45.970897 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:45.970870 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b7bml"] Oct 30 03:57:46.017027 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.017008 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58706aa4-9bad-4e69-abdb-b9af8390f27c-config-volume\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.017027 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.017033 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jsmr\" (UniqueName: \"kubernetes.io/projected/58706aa4-9bad-4e69-abdb-b9af8390f27c-kube-api-access-2jsmr\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.017149 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.017053 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fda4935-bef9-407b-9ef9-ea0725f519f2-cert\") pod \"ingress-canary-wq96k\" (UID: \"0fda4935-bef9-407b-9ef9-ea0725f519f2\") " pod="openshift-ingress-canary/ingress-canary-wq96k" Oct 30 03:57:46.017149 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.017084 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58706aa4-9bad-4e69-abdb-b9af8390f27c-tmp-dir\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.017213 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.017158 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58706aa4-9bad-4e69-abdb-b9af8390f27c-metrics-tls\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.017244 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.017218 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z72tk\" (UniqueName: \"kubernetes.io/projected/0fda4935-bef9-407b-9ef9-ea0725f519f2-kube-api-access-z72tk\") pod \"ingress-canary-wq96k\" (UID: \"0fda4935-bef9-407b-9ef9-ea0725f519f2\") " pod="openshift-ingress-canary/ingress-canary-wq96k" Oct 30 03:57:46.057442 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.057381 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-846td\"" Oct 30 03:57:46.057507 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.057446 2570 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 03:57:46.058276 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.058263 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 03:57:46.058402 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.058391 2570 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 03:57:46.117688 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.117665 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z72tk\" (UniqueName: \"kubernetes.io/projected/0fda4935-bef9-407b-9ef9-ea0725f519f2-kube-api-access-z72tk\") pod \"ingress-canary-wq96k\" (UID: \"0fda4935-bef9-407b-9ef9-ea0725f519f2\") " pod="openshift-ingress-canary/ingress-canary-wq96k" Oct 30 03:57:46.117759 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.117698 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4d24ab0e-8485-45c0-8b05-a149634357ff-crio-socket\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.117759 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.117722 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvzqm\" (UniqueName: \"kubernetes.io/projected/4d24ab0e-8485-45c0-8b05-a149634357ff-kube-api-access-vvzqm\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.117832 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.117785 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d24ab0e-8485-45c0-8b05-a149634357ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.117910 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.117886 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d24ab0e-8485-45c0-8b05-a149634357ff-data-volume\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.117939 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.117908 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58706aa4-9bad-4e69-abdb-b9af8390f27c-config-volume\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.117939 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.117923 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jsmr\" (UniqueName: \"kubernetes.io/projected/58706aa4-9bad-4e69-abdb-b9af8390f27c-kube-api-access-2jsmr\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.117996 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.117961 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d24ab0e-8485-45c0-8b05-a149634357ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.117996 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.117988 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fda4935-bef9-407b-9ef9-ea0725f519f2-cert\") pod \"ingress-canary-wq96k\" (UID: \"0fda4935-bef9-407b-9ef9-ea0725f519f2\") " pod="openshift-ingress-canary/ingress-canary-wq96k" Oct 30 03:57:46.118136 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.118123 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58706aa4-9bad-4e69-abdb-b9af8390f27c-tmp-dir\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.118172 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.118145 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58706aa4-9bad-4e69-abdb-b9af8390f27c-metrics-tls\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.118472 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.118452 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58706aa4-9bad-4e69-abdb-b9af8390f27c-tmp-dir\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.118603 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.118586 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58706aa4-9bad-4e69-abdb-b9af8390f27c-config-volume\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.121737 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.121716 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58706aa4-9bad-4e69-abdb-b9af8390f27c-metrics-tls\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.121842 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.121766 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fda4935-bef9-407b-9ef9-ea0725f519f2-cert\") pod \"ingress-canary-wq96k\" (UID: \"0fda4935-bef9-407b-9ef9-ea0725f519f2\") " pod="openshift-ingress-canary/ingress-canary-wq96k" Oct 30 03:57:46.151529 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.151505 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jsmr\" (UniqueName: \"kubernetes.io/projected/58706aa4-9bad-4e69-abdb-b9af8390f27c-kube-api-access-2jsmr\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.151837 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.151816 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z72tk\" (UniqueName: \"kubernetes.io/projected/0fda4935-bef9-407b-9ef9-ea0725f519f2-kube-api-access-z72tk\") pod \"ingress-canary-wq96k\" (UID: \"0fda4935-bef9-407b-9ef9-ea0725f519f2\") " pod="openshift-ingress-canary/ingress-canary-wq96k" Oct 30 03:57:46.155493 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.155475 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:46.157547 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.157530 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wq96k" Oct 30 03:57:46.220962 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.220932 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4d24ab0e-8485-45c0-8b05-a149634357ff-crio-socket\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.221063 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.221004 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvzqm\" (UniqueName: \"kubernetes.io/projected/4d24ab0e-8485-45c0-8b05-a149634357ff-kube-api-access-vvzqm\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.221201 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.221187 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4d24ab0e-8485-45c0-8b05-a149634357ff-crio-socket\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.221530 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.221510 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d24ab0e-8485-45c0-8b05-a149634357ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.221994 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.221973 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d24ab0e-8485-45c0-8b05-a149634357ff-data-volume\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.222085 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.222048 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d24ab0e-8485-45c0-8b05-a149634357ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.222411 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.222384 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d24ab0e-8485-45c0-8b05-a149634357ff-data-volume\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.222659 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.222641 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d24ab0e-8485-45c0-8b05-a149634357ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.232290 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.232264 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d24ab0e-8485-45c0-8b05-a149634357ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.244447 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.244408 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvzqm\" (UniqueName: \"kubernetes.io/projected/4d24ab0e-8485-45c0-8b05-a149634357ff-kube-api-access-vvzqm\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.268745 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.268730 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:46.475684 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.475662 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wq96k"] Oct 30 03:57:46.478257 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:46.478210 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fda4935_bef9_407b_9ef9_ea0725f519f2.slice/crio-4fe46aaf172fc3c4e140fde06e9f5a11e5f35ae15d821a71324c1db5589b9ed3 WatchSource:0}: Error finding container 4fe46aaf172fc3c4e140fde06e9f5a11e5f35ae15d821a71324c1db5589b9ed3: Status 404 returned error can't find the container with id 4fe46aaf172fc3c4e140fde06e9f5a11e5f35ae15d821a71324c1db5589b9ed3 Oct 30 03:57:46.480635 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.480613 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zrv7n"] Oct 30 03:57:46.482113 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:46.482093 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58706aa4_9bad_4e69_abdb_b9af8390f27c.slice/crio-ae9e5f64487b9cf647c340cd997cbd4053cccf6cd4c3ad1436f51171cd28a9cf WatchSource:0}: Error finding container ae9e5f64487b9cf647c340cd997cbd4053cccf6cd4c3ad1436f51171cd28a9cf: Status 404 returned error can't find the container with id ae9e5f64487b9cf647c340cd997cbd4053cccf6cd4c3ad1436f51171cd28a9cf Oct 30 03:57:46.564077 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:46.564039 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b7bml"] Oct 30 03:57:46.565592 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:46.565562 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d24ab0e_8485_45c0_8b05_a149634357ff.slice/crio-cb0599709a1b704df4591cc726efdc7ef213d151d80052bcc8727d3832aa53e4 WatchSource:0}: Error finding container cb0599709a1b704df4591cc726efdc7ef213d151d80052bcc8727d3832aa53e4: Status 404 returned error can't find the container with id cb0599709a1b704df4591cc726efdc7ef213d151d80052bcc8727d3832aa53e4 Oct 30 03:57:47.276100 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:47.275925 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wq96k" event={"ID":"0fda4935-bef9-407b-9ef9-ea0725f519f2","Type":"ContainerStarted","Data":"4fe46aaf172fc3c4e140fde06e9f5a11e5f35ae15d821a71324c1db5589b9ed3"} Oct 30 03:57:47.277780 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:47.277750 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zrv7n" event={"ID":"58706aa4-9bad-4e69-abdb-b9af8390f27c","Type":"ContainerStarted","Data":"ae9e5f64487b9cf647c340cd997cbd4053cccf6cd4c3ad1436f51171cd28a9cf"} Oct 30 03:57:47.279737 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:47.279717 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7bml" event={"ID":"4d24ab0e-8485-45c0-8b05-a149634357ff","Type":"ContainerStarted","Data":"98c827dc4c61b5b175515916c8a0778c5d5fc6e794cd68fd05b3301e1dc4effd"} Oct 30 03:57:47.279827 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:47.279764 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7bml" event={"ID":"4d24ab0e-8485-45c0-8b05-a149634357ff","Type":"ContainerStarted","Data":"cb0599709a1b704df4591cc726efdc7ef213d151d80052bcc8727d3832aa53e4"} Oct 30 03:57:48.274348 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:48.274312 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:48.286766 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:48.286722 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7bml" event={"ID":"4d24ab0e-8485-45c0-8b05-a149634357ff","Type":"ContainerStarted","Data":"b4dc5d7e96024fa3afb809f21096ab022e819e2fdebd18f2392515d1c770c00f"} Oct 30 03:57:48.510989 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:48.510944 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:49.290315 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:49.290252 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zrv7n" event={"ID":"58706aa4-9bad-4e69-abdb-b9af8390f27c","Type":"ContainerStarted","Data":"ae1b3f28ca8c89430e38ea0f490f454c1e5ad8203c44464872d72f4fb9a84918"} Oct 30 03:57:49.290315 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:49.290297 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zrv7n" event={"ID":"58706aa4-9bad-4e69-abdb-b9af8390f27c","Type":"ContainerStarted","Data":"98113344a6888654e73a3e6b4267ed7efa9ed860234619de696d826638054f9a"} Oct 30 03:57:49.290695 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:49.290328 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:49.345349 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:49.345300 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-zrv7n" podStartSLOduration=2.133809302 podStartE2EDuration="4.345287052s" totalImagesPullingTime="2.21147775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:49.344703835 +0000 UTC m=+32.857367521" watchObservedRunningTime="2026-10-30 03:57:49.345287052 +0000 UTC m=+32.857950717" Oct 30 03:57:49.722540 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:49.722513 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:49.990828 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:49.990750 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:50.294629 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:50.294548 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wq96k" event={"ID":"0fda4935-bef9-407b-9ef9-ea0725f519f2","Type":"ContainerStarted","Data":"53167d2a3999df98126971fb43f50969bef36b267dc0b48ab9eab848febc83db"} Oct 30 03:57:50.296443 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:50.296401 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b7bml" event={"ID":"4d24ab0e-8485-45c0-8b05-a149634357ff","Type":"ContainerStarted","Data":"00ca84f78ea51f7650d4724d6d7816594b1f3e3bc4f5d9134557afb57a398d8d"} Oct 30 03:57:50.339234 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:50.339195 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wq96k" podStartSLOduration=2.065533885 podStartE2EDuration="5.33918247s" totalImagesPullingTime="3.273648585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:50.338683195 +0000 UTC m=+33.851346862" watchObservedRunningTime="2026-10-30 03:57:50.33918247 +0000 UTC m=+33.851846136" Oct 30 03:57:50.385985 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:50.385946 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b7bml" podStartSLOduration=2.362741409 podStartE2EDuration="5.385932382s" totalImagesPullingTime="3.023190973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:57:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:50.385535044 +0000 UTC m=+33.898198709" watchObservedRunningTime="2026-10-30 03:57:50.385932382 +0000 UTC m=+33.898596048" Oct 30 03:57:50.760169 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:50.760139 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:50.760321 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:50.760195 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:50.763156 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:50.763132 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:50.763292 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:50.763139 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:50.860461 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:50.860432 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrmh\" (UniqueName: \"kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh\") pod \"network-check-target-v5zp7\" (UID: \"438677a3-904a-474e-bbed-4032260a3a3c\") " pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:50.862750 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:50.862728 2570 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrmh\" (UniqueName: \"kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh\") pod \"network-check-target-v5zp7\" (UID: \"438677a3-904a-474e-bbed-4032260a3a3c\") " pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:51.047054 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:51.046975 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:51.052124 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:51.052099 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:51.060252 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:51.060230 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:51.356547 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:51.356523 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-v5zp7"] Oct 30 03:57:51.358021 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:51.357996 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438677a3_904a_474e_bbed_4032260a3a3c.slice/crio-3f37f9ad9ba315f552b7c4d2625b92b253a54e0cdb45de54ff509d158c4fa9ca WatchSource:0}: Error finding container 3f37f9ad9ba315f552b7c4d2625b92b253a54e0cdb45de54ff509d158c4fa9ca: Status 404 returned error can't find the container with id 3f37f9ad9ba315f552b7c4d2625b92b253a54e0cdb45de54ff509d158c4fa9ca Oct 30 03:57:51.386017 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:51.385997 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p2mmh"] Oct 30 03:57:51.387348 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:51.387320 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93cbeaeb_857f_4c3f_825f_62c46320edd0.slice/crio-e2288c6b2aff49155d3a73d26b5ec61a3c4ea316a699466006c19c68dea61d6b WatchSource:0}: Error finding container e2288c6b2aff49155d3a73d26b5ec61a3c4ea316a699466006c19c68dea61d6b: Status 404 returned error can't find the container with id e2288c6b2aff49155d3a73d26b5ec61a3c4ea316a699466006c19c68dea61d6b Oct 30 03:57:51.410071 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:51.410054 2570 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tffth"] Oct 30 03:57:51.411296 ip-10-0-138-177 kubenswrapper[2570]: W1030 03:57:51.411201 2570 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70c8827_8921_4231_8144_395f162320c2.slice/crio-abbab493d5f0878d5c4b363947ca2b72eaa617147509ef013c6f276f07121baa WatchSource:0}: Error finding container abbab493d5f0878d5c4b363947ca2b72eaa617147509ef013c6f276f07121baa: Status 404 returned error can't find the container with id abbab493d5f0878d5c4b363947ca2b72eaa617147509ef013c6f276f07121baa Oct 30 03:57:52.304916 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:52.304836 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tffth" event={"ID":"e70c8827-8921-4231-8144-395f162320c2","Type":"ContainerStarted","Data":"abbab493d5f0878d5c4b363947ca2b72eaa617147509ef013c6f276f07121baa"} Oct 30 03:57:52.307309 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:52.307276 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p2mmh" event={"ID":"93cbeaeb-857f-4c3f-825f-62c46320edd0","Type":"ContainerStarted","Data":"e2288c6b2aff49155d3a73d26b5ec61a3c4ea316a699466006c19c68dea61d6b"} Oct 30 03:57:52.308602 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:52.308574 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v5zp7" event={"ID":"438677a3-904a-474e-bbed-4032260a3a3c","Type":"ContainerStarted","Data":"3f37f9ad9ba315f552b7c4d2625b92b253a54e0cdb45de54ff509d158c4fa9ca"} Oct 30 03:57:52.810882 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:52.810836 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:53.075381 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:53.075264 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:53.315331 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:53.315292 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tffth" event={"ID":"e70c8827-8921-4231-8144-395f162320c2","Type":"ContainerStarted","Data":"0dcfbbc29cbd263270bfbcdc527618c040f326f8cbeecfe5866a3ed81e20da29"} Oct 30 03:57:53.315512 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:53.315365 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tffth" event={"ID":"e70c8827-8921-4231-8144-395f162320c2","Type":"ContainerStarted","Data":"754ec1b3c8e9cbb60d23e8d32ea3a010f46bae72016da3d8328ed7cb15ef6064"} Oct 30 03:57:53.344880 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:53.344796 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:53.356753 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:53.356702 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tffth" podStartSLOduration=34.847763992 podStartE2EDuration="36.356687741s" totalImagesPullingTime="1.508923749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:53.356199257 +0000 UTC m=+36.868862933" watchObservedRunningTime="2026-10-30 03:57:53.356687741 +0000 UTC m=+36.869351411" Oct 30 03:57:53.566471 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:53.566427 2570 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:55.324156 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:55.324120 2570 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-v5zp7" event={"ID":"438677a3-904a-474e-bbed-4032260a3a3c","Type":"ContainerStarted","Data":"f91c8b6d24eea59f1927de3164a8e51487398926ffe9700863ff5ca52af04d42"} Oct 30 03:57:55.324688 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:55.324254 2570 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:55.365783 ip-10-0-138-177 kubenswrapper[2570]: I1030 03:57:55.365729 2570 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-v5zp7" podStartSLOduration=34.903956382 podStartE2EDuration="38.365711393s" totalImagesPullingTime="3.461755011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:57:55.365236944 +0000 UTC m=+38.877900611" watchObservedRunningTime="2026-10-30 03:57:55.365711393 +0000 UTC m=+38.878375064" Oct 30 03:57:56.655207 ip-10-0-138-177 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 03:57:56.667865 ip-10-0-138-177 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 03:57:56.668029 ip-10-0-138-177 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 03:57:56.668186 ip-10-0-138-177 systemd[1]: kubelet.service: Consumed 2.483s CPU time. Oct 30 03:57:56.679091 ip-10-0-138-177 systemd[1]: Starting Kubernetes Kubelet... Oct 30 03:57:56.757192 ip-10-0-138-177 kubenswrapper[4793]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:57:56.757192 ip-10-0-138-177 kubenswrapper[4793]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 03:57:56.757192 ip-10-0-138-177 kubenswrapper[4793]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:57:56.757192 ip-10-0-138-177 kubenswrapper[4793]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 03:57:56.759186 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759114 4793 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:56.759186 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759185 4793 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759190 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759193 4793 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759196 4793 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759200 4793 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759202 4793 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759205 4793 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759208 4793 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759210 4793 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759213 4793 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759215 4793 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759218 4793 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759221 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759223 4793 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759226 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759229 4793 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759231 4793 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759234 4793 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759236 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:56.759255 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759243 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759246 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759249 4793 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759252 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759254 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759257 4793 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759260 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759263 4793 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759266 4793 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759269 4793 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759272 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759274 4793 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759277 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759280 4793 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759282 4793 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759285 4793 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759287 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759290 4793 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759293 4793 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759296 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:56.759753 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759299 4793 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759301 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759304 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759306 4793 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759309 4793 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759311 4793 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759314 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759316 4793 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759319 4793 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759322 4793 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759324 4793 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759327 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759329 4793 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759332 4793 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759335 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759337 4793 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759341 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759344 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759348 4793 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:56.760268 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759352 4793 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759355 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759365 4793 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759369 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759373 4793 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759375 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759378 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759380 4793 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759383 4793 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759386 4793 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759389 4793 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759391 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759394 4793 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759405 4793 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759408 4793 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759411 4793 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759432 4793 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759435 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759438 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:56.760767 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759442 4793 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759446 4793 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759449 4793 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759452 4793 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759455 4793 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759459 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759462 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759465 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759467 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759470 4793 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759472 4793 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759475 4793 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759478 4793 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759480 4793 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759483 4793 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759485 4793 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759488 4793 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759491 4793 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759494 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759496 4793 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:56.761236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759499 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759501 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759504 4793 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759506 4793 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759509 4793 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759511 4793 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759514 4793 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.759518 4793 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760254 4793 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760261 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760263 4793 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760266 4793 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760269 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760272 4793 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760275 4793 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760278 4793 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760281 4793 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760284 4793 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760286 4793 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:56.761756 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760289 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760292 4793 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760295 4793 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760297 4793 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760300 4793 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760302 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760305 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760308 4793 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760310 4793 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760313 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760315 4793 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760319 4793 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760321 4793 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760324 4793 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760326 4793 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760329 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760332 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760334 4793 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760337 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760339 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:56.762243 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760342 4793 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760345 4793 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760348 4793 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760350 4793 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760353 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760355 4793 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760358 4793 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760360 4793 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760363 4793 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760365 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760368 4793 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760370 4793 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760373 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760375 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760378 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760380 4793 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760383 4793 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760385 4793 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760389 4793 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:56.762816 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760391 4793 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760394 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760397 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760399 4793 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760402 4793 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760405 4793 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760407 4793 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760410 4793 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760412 4793 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760431 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760435 4793 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760438 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760441 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760444 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760446 4793 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760449 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760452 4793 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760455 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760458 4793 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:56.763295 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760460 4793 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760463 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760466 4793 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760468 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760471 4793 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760473 4793 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760476 4793 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760478 4793 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760481 4793 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760484 4793 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760486 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760489 4793 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760491 4793 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760494 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760497 4793 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760500 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760502 4793 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760505 4793 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760509 4793 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:56.763825 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760513 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760517 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760519 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760522 4793 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760524 4793 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760529 4793 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760532 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760535 4793 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760538 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760541 4793 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760545 4793 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760547 4793 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760550 4793 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760553 4793 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760555 4793 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760558 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760561 4793 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.760563 4793 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761046 4793 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761055 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:56.764296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761060 4793 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761063 4793 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761066 4793 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761069 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761072 4793 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761075 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761077 4793 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761080 4793 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761083 4793 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761085 4793 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761088 4793 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761090 4793 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761093 4793 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761095 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761098 4793 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761101 4793 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761105 4793 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761109 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761112 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:56.764978 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761115 4793 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761117 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761120 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761125 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761128 4793 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761130 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761133 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761135 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761138 4793 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761141 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761143 4793 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761146 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761151 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761153 4793 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761156 4793 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761158 4793 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761161 4793 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761163 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761166 4793 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761168 4793 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:56.765480 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761171 4793 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761174 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761176 4793 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761179 4793 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761181 4793 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761184 4793 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761186 4793 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761188 4793 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761191 4793 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761194 4793 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761196 4793 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761198 4793 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761201 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761203 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761206 4793 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761210 4793 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761213 4793 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761216 4793 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761218 4793 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:56.766336 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761221 4793 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761223 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761226 4793 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761228 4793 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761231 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761234 4793 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761237 4793 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761240 4793 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761242 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761245 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761250 4793 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761253 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761256 4793 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761258 4793 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761261 4793 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761263 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761266 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761268 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761271 4793 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:56.766901 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761273 4793 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761276 4793 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761278 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761281 4793 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761283 4793 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761286 4793 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761288 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761291 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761293 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761296 4793 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761299 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761302 4793 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761305 4793 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761307 4793 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761310 4793 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761312 4793 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761314 4793 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761317 4793 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761321 4793 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761323 4793 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:56.767389 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761325 4793 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761328 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761330 4793 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761333 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761335 4793 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761338 4793 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.761340 4793 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761448 4793 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761456 4793 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761463 4793 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761467 4793 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761471 4793 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761475 4793 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761482 4793 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761486 4793 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761489 4793 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761492 4793 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761496 4793 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761500 4793 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761503 4793 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761506 4793 flags.go:64] FLAG: --cgroup-root="" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761509 4793 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761512 4793 flags.go:64] FLAG: --client-ca-file="" Oct 30 03:57:56.767954 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761515 4793 flags.go:64] FLAG: --cloud-provider="external" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761519 4793 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761523 4793 flags.go:64] FLAG: --cluster-domain="" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761525 4793 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761529 4793 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761532 4793 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761535 4793 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761540 4793 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761543 4793 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761546 4793 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761549 4793 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761552 4793 flags.go:64] FLAG: --contention-profiling="false" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761555 4793 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761559 4793 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761562 4793 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761565 4793 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761569 4793 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761572 4793 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761575 4793 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761578 4793 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761581 4793 flags.go:64] FLAG: --enable-server="true" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761584 4793 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761589 4793 flags.go:64] FLAG: --event-burst="100" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761592 4793 flags.go:64] FLAG: --event-qps="50" Oct 30 03:57:56.768549 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761595 4793 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761598 4793 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761601 4793 flags.go:64] FLAG: --eviction-hard="" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761605 4793 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761609 4793 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761612 4793 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761615 4793 flags.go:64] FLAG: --eviction-soft="" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761618 4793 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761621 4793 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761624 4793 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761627 4793 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761630 4793 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761633 4793 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761636 4793 flags.go:64] FLAG: --feature-gates="" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761640 4793 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761643 4793 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761647 4793 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761650 4793 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761653 4793 flags.go:64] FLAG: --healthz-port="10248" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761656 4793 flags.go:64] FLAG: --help="false" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761659 4793 flags.go:64] FLAG: --hostname-override="ip-10-0-138-177.ec2.internal" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761662 4793 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761665 4793 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 03:57:56.769154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761668 4793 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761672 4793 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761675 4793 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761678 4793 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761681 4793 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761684 4793 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761687 4793 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761690 4793 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761693 4793 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761696 4793 flags.go:64] FLAG: --kube-reserved="" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761699 4793 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761702 4793 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761705 4793 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761708 4793 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761711 4793 flags.go:64] FLAG: --lock-file="" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761713 4793 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761716 4793 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761719 4793 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761725 4793 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761728 4793 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761731 4793 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761735 4793 flags.go:64] FLAG: --logging-format="text" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761738 4793 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761741 4793 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 03:57:56.769773 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761744 4793 flags.go:64] FLAG: --manifest-url="" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761747 4793 flags.go:64] FLAG: --manifest-url-header="" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761751 4793 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761754 4793 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761758 4793 flags.go:64] FLAG: --max-pods="110" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761761 4793 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761764 4793 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761767 4793 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761770 4793 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761773 4793 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761776 4793 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761779 4793 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761785 4793 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761791 4793 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761794 4793 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761797 4793 flags.go:64] FLAG: --pod-cidr="" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761800 4793 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761803 4793 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761807 4793 flags.go:64] FLAG: --pods-per-core="0" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761810 4793 flags.go:64] FLAG: --port="10250" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761813 4793 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761816 4793 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01e5b2c5e304b54b7" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761819 4793 flags.go:64] FLAG: --qos-reserved="" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761822 4793 flags.go:64] FLAG: --read-only-port="10255" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761825 4793 flags.go:64] FLAG: --register-node="true" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761829 4793 flags.go:64] FLAG: --register-with-taints="" Oct 30 03:57:56.770367 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761832 4793 flags.go:64] FLAG: --registry-burst="10" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761835 4793 flags.go:64] FLAG: --registry-qps="5" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761838 4793 flags.go:64] FLAG: --reserved-cpus="" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761842 4793 flags.go:64] FLAG: --reserved-memory="" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761846 4793 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761849 4793 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761852 4793 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761855 4793 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761858 4793 flags.go:64] FLAG: --runonce="false" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761861 4793 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761864 4793 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761867 4793 flags.go:64] FLAG: --seccomp-default="false" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761870 4793 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761873 4793 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761876 4793 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761879 4793 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761882 4793 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761886 4793 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761889 4793 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761894 4793 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761897 4793 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761900 4793 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761903 4793 flags.go:64] FLAG: --system-cgroups="" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761906 4793 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761912 4793 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 03:57:56.771043 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761915 4793 flags.go:64] FLAG: --tls-cert-file="" Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761918 4793 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761922 4793 flags.go:64] FLAG: --tls-min-version="" Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761925 4793 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761928 4793 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761931 4793 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761934 4793 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761937 4793 flags.go:64] FLAG: --v="2" Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761941 4793 flags.go:64] FLAG: --version="false" Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761945 4793 flags.go:64] FLAG: --vmodule="" Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761949 4793 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.761954 4793 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762094 4793 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762098 4793 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762101 4793 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762105 4793 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762108 4793 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762111 4793 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762114 4793 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762117 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762119 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762122 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:56.771695 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762126 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762129 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762131 4793 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762134 4793 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762138 4793 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762140 4793 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762143 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762146 4793 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762149 4793 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762153 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762156 4793 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762158 4793 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762161 4793 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762164 4793 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762166 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762169 4793 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762172 4793 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762174 4793 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762177 4793 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762180 4793 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:56.772411 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762183 4793 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762185 4793 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762188 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762191 4793 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762194 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762197 4793 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762199 4793 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762202 4793 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762204 4793 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762208 4793 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762210 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762213 4793 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762215 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762218 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762220 4793 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762223 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762227 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762230 4793 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762232 4793 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762235 4793 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:56.773183 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762238 4793 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762243 4793 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762245 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762248 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762251 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762254 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762257 4793 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762259 4793 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762262 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762265 4793 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762268 4793 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762270 4793 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762273 4793 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762276 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762278 4793 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762281 4793 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762284 4793 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762287 4793 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762289 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:56.773984 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762292 4793 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762295 4793 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762297 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762300 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762302 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762305 4793 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762308 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762310 4793 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762313 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762317 4793 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762319 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762322 4793 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762325 4793 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762327 4793 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762332 4793 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762334 4793 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762337 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762340 4793 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762344 4793 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762348 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:56.774479 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762351 4793 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762354 4793 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762357 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762359 4793 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762362 4793 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762365 4793 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762367 4793 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762370 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762373 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762376 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762381 4793 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762384 4793 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762387 4793 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762390 4793 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762393 4793 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762395 4793 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:56.775064 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762398 4793 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.762439 4793 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762474 4793 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762478 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762481 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762483 4793 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762488 4793 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762491 4793 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762493 4793 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762496 4793 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762499 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762502 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762505 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762508 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762510 4793 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:56.775615 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762513 4793 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762516 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762518 4793 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762521 4793 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762523 4793 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762526 4793 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762528 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762531 4793 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762534 4793 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762536 4793 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762539 4793 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762542 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762545 4793 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762547 4793 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762550 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762552 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762555 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762557 4793 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762560 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762562 4793 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:56.776177 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762565 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762567 4793 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762570 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762573 4793 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762576 4793 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762579 4793 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762582 4793 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762585 4793 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762588 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762590 4793 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762595 4793 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762598 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762602 4793 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762605 4793 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762607 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762610 4793 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762612 4793 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762615 4793 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762618 4793 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:56.776698 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762620 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762623 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762625 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762629 4793 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762632 4793 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762636 4793 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762638 4793 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762641 4793 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762643 4793 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762646 4793 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762649 4793 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762651 4793 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762653 4793 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762656 4793 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762658 4793 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762661 4793 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762665 4793 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762667 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762670 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:56.777161 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762673 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762676 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762679 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762682 4793 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762684 4793 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762687 4793 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762689 4793 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762692 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762694 4793 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762697 4793 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762700 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762703 4793 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762705 4793 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762708 4793 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762711 4793 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762713 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762716 4793 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762718 4793 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762721 4793 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762724 4793 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:56.777646 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762726 4793 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762729 4793 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762732 4793 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762734 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762737 4793 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762739 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762742 4793 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762744 4793 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762747 4793 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762752 4793 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762755 4793 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762757 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762760 4793 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762764 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.762766 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.767592 4793 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.767609 4793 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.767627 4793 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.767631 4793 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767688 4793 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:56.778187 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767693 4793 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767697 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767700 4793 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767703 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767706 4793 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767708 4793 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767711 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767714 4793 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767717 4793 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767719 4793 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767722 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767725 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767728 4793 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767730 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767733 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767737 4793 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767741 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767746 4793 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767750 4793 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:56.778835 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767770 4793 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767778 4793 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767782 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767785 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767788 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767792 4793 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767794 4793 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767797 4793 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767801 4793 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767804 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767806 4793 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767809 4793 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767812 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767814 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767817 4793 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767819 4793 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767822 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767825 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767827 4793 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767830 4793 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:56.779650 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767833 4793 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767835 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767838 4793 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767840 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767844 4793 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767846 4793 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767849 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767852 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767855 4793 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767858 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767861 4793 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767863 4793 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767866 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767868 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767871 4793 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767873 4793 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767876 4793 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767879 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767881 4793 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767884 4793 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:56.780500 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767887 4793 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767890 4793 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767892 4793 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767895 4793 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767898 4793 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767901 4793 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767903 4793 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767906 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767908 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767911 4793 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767913 4793 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767916 4793 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767918 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767921 4793 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767923 4793 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767926 4793 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767929 4793 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767932 4793 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767935 4793 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767938 4793 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:56.781484 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767941 4793 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767943 4793 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767946 4793 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767948 4793 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767950 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767953 4793 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767956 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767958 4793 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767961 4793 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767964 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767967 4793 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767970 4793 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767972 4793 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767975 4793 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767979 4793 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767983 4793 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767986 4793 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767989 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767991 4793 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:56.782382 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767994 4793 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767997 4793 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.767999 4793 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768002 4793 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768004 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768007 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768010 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.768040 4793 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768197 4793 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768201 4793 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768204 4793 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768207 4793 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768209 4793 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768213 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768215 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768218 4793 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:56.783264 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768221 4793 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768223 4793 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768226 4793 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768228 4793 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768231 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768233 4793 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768236 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768239 4793 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768241 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768244 4793 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768246 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768249 4793 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768251 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768254 4793 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768257 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768259 4793 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768262 4793 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768264 4793 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768267 4793 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768269 4793 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:56.784004 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768272 4793 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768275 4793 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768277 4793 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768280 4793 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768282 4793 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768285 4793 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768287 4793 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768290 4793 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768294 4793 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768296 4793 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768299 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768303 4793 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768308 4793 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768311 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768314 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768317 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768320 4793 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768323 4793 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768326 4793 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768329 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:56.784885 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768331 4793 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768334 4793 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768336 4793 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768339 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768342 4793 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768344 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768347 4793 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768349 4793 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768352 4793 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768355 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768357 4793 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768360 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768362 4793 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768365 4793 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768368 4793 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768370 4793 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768373 4793 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768375 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768377 4793 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768380 4793 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:56.785697 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768383 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768385 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768387 4793 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768391 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768394 4793 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768397 4793 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768399 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768402 4793 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768405 4793 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768408 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768410 4793 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768413 4793 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768431 4793 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768435 4793 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768437 4793 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768440 4793 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768443 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768445 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768448 4793 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:56.786323 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768451 4793 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768454 4793 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768456 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768459 4793 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768461 4793 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768465 4793 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768469 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768471 4793 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768474 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768476 4793 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768479 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768482 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768485 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768487 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768490 4793 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768492 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768495 4793 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768502 4793 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:56.786803 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768505 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.768530 4793 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768564 4793 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768568 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768571 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768575 4793 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768578 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768582 4793 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768585 4793 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768588 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768591 4793 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768594 4793 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768597 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768600 4793 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768602 4793 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768605 4793 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 03:57:56.787236 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768608 4793 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768610 4793 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768614 4793 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768617 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768620 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768623 4793 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768626 4793 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768628 4793 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768631 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768634 4793 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768636 4793 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768640 4793 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768643 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768646 4793 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768649 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768652 4793 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768656 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768659 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768662 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 03:57:56.787783 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768664 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768667 4793 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768669 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768672 4793 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768675 4793 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768678 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768680 4793 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768683 4793 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768685 4793 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768688 4793 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768691 4793 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768693 4793 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768696 4793 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768699 4793 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768701 4793 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768705 4793 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768708 4793 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768710 4793 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768713 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768715 4793 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 03:57:56.788296 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768718 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768721 4793 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768723 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768726 4793 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768728 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768731 4793 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768734 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768736 4793 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768739 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768742 4793 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768747 4793 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768749 4793 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768752 4793 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768754 4793 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768757 4793 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768759 4793 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768762 4793 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768765 4793 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768767 4793 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768770 4793 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 03:57:56.788900 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768772 4793 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768776 4793 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768779 4793 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768781 4793 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768784 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768788 4793 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768792 4793 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768797 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768800 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768803 4793 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768806 4793 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768809 4793 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768813 4793 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768815 4793 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768818 4793 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768821 4793 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768823 4793 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768826 4793 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768829 4793 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 03:57:56.789710 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768831 4793 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768834 4793 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768838 4793 feature_gate.go:386] unrecognized feature gate: Example Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768841 4793 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768844 4793 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768846 4793 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768849 4793 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768851 4793 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768854 4793 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768857 4793 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768859 4793 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768863 4793 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768866 4793 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:57:56.768868 4793 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.768922 4793 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.769874 4793 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.769926 4793 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.770105 4793 server.go:1014] "Starting client certificate rotation" Oct 30 03:57:56.790494 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.770249 4793 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 03:57:56.791351 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.770300 4793 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 03:52:16 +0000 UTC" deadline="2028-05-07 23:21:50.431955835 +0000 UTC" Oct 30 03:57:56.791351 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.770326 4793 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13339h23m53.661631613s" Oct 30 03:57:56.791351 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.770556 4793 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:57:56.791351 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.771745 4793 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 03:57:56.791351 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.773298 4793 log.go:25] "Validated CRI v1 runtime API" Oct 30 03:57:56.791351 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.774952 4793 log.go:25] "Validated CRI v1 image API" Oct 30 03:57:56.791351 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.775735 4793 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 03:57:56.791351 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.777331 4793 fs.go:135] Filesystem UUIDs: map[6289d932-d5da-45bf-bb60-ee9eb419b2ba:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9e810927-6396-4172-a431-27de3cc6f8e1:/dev/nvme0n1p3] Oct 30 03:57:56.791790 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.777349 4793 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/004d1a87951da880edfce6abcf22b6583af05d032b6b9e1af79ea47b03f5fc85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/004d1a87951da880edfce6abcf22b6583af05d032b6b9e1af79ea47b03f5fc85/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c0ddd85ce8284af9f2e2b438c86be40ea67764495e5ceaa1fab27d8513cf984/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c0ddd85ce8284af9f2e2b438c86be40ea67764495e5ceaa1fab27d8513cf984/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3f37f9ad9ba315f552b7c4d2625b92b253a54e0cdb45de54ff509d158c4fa9ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3f37f9ad9ba315f552b7c4d2625b92b253a54e0cdb45de54ff509d158c4fa9ca/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44b8082218dde0674055a08592a46b04f75ca8da879155d82b9f460e35a83f6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44b8082218dde0674055a08592a46b04f75ca8da879155d82b9f460e35a83f6c/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4943309beae3fa168b16fa6651ee168faea5dabc32e9fa322e8de13241bd72aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4943309beae3fa168b16fa6651ee168faea5dabc32e9fa322e8de13241bd72aa/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fe46aaf172fc3c4e140fde06e9f5a11e5f35ae15d821a71324c1db5589b9ed3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fe46aaf172fc3c4e140fde06e9f5a11e5f35ae15d821a71324c1db5589b9ed3/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/62b0c2bb8e22a9ee06b1009118d960ea8359a4b723d4e8a349590afee949810d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/62b0c2bb8e22a9ee06b1009118d960ea8359a4b723d4e8a349590afee949810d/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abbab493d5f0878d5c4b363947ca2b72eaa617147509ef013c6f276f07121baa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abbab493d5f0878d5c4b363947ca2b72eaa617147509ef013c6f276f07121baa/userdata/shm major:0 minor:274 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae5cb5e8f5b97a78add2edabad7c2e76ae5fa5c3fe25ef9df4910ca1bd2045d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae5cb5e8f5b97a78add2edabad7c2e76ae5fa5c3fe25ef9df4910ca1bd2045d3/userdata/shm major:0 minor:120 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae9e5f64487b9cf647c340cd997cbd4053cccf6cd4c3ad1436f51171cd28a9cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae9e5f64487b9cf647c340cd997cbd4053cccf6cd4c3ad1436f51171cd28a9cf/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aee539f33e518dc52781f4c17bd7d1a48147663fcd223b756f32a58cf8a17e7c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aee539f33e518dc52781f4c17bd7d1a48147663fcd223b756f32a58cf8a17e7c/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b323e055211f8138d7a3042ac59e64b439fc2fe9726d4403697c69548f827853/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b323e055211f8138d7a3042ac59e64b439fc2fe9726d4403697c69548f827853/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b53ba5682503e82dce02a1cf8ab9edb1e81f2ca6125a5f2e139764f496e5bd14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b53ba5682503e82dce02a1cf8ab9edb1e81f2ca6125a5f2e139764f496e5bd14/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1befeb0a3b86351b7c14a4e2e2d45c17eecbd5b0b65949d1d0be4616abcc6fb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1befeb0a3b86351b7c14a4e2e2d45c17eecbd5b0b65949d1d0be4616abcc6fb/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cb0599709a1b704df4591cc726efdc7ef213d151d80052bcc8727d3832aa53e4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cb0599709a1b704df4591cc726efdc7ef213d151d80052bcc8727d3832aa53e4/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e2288c6b2aff49155d3a73d26b5ec61a3c4ea316a699466006c19c68dea61d6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e2288c6b2aff49155d3a73d26b5ec61a3c4ea316a699466006c19c68dea61d6b/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9e3a029cd0b30c569bc987d45c211a622d84cbe6a04cf31e8ec796f51f14273/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9e3a029cd0b30c569bc987d45c211a622d84cbe6a04cf31e8ec796f51f14273/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fda4935-bef9-407b-9ef9-ea0725f519f2/volumes/kubernetes.io~projected/kube-api-access-z72tk:{mountpoint:/var/lib/kubelet/pods/0fda4935-bef9-407b-9ef9-ea0725f519f2/volumes/kubernetes.io~projected/kube-api-access-z72tk major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fda4935-bef9-407b-9ef9-ea0725f519f2/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0fda4935-bef9-407b-9ef9-ea0725f519f2/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c471f71-4a76-4003-9f66-663b894892c0/volumes/kubernetes.io~projected/kube-api-access-bns9g:{mountpoint:/var/lib/kubelet/pods/1c471f71-4a76-4003-9f66-663b894892c0/volumes/kubernetes.io~projected/kube-api-access-bns9g major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/438677a3-904a-474e-bbed-4032260a3a3c/volumes/kubernetes.io~projected/kube-api-access-4lrmh:{mountpoint:/var/lib/kubelet/pods/438677a3-904a-474e-bbed-4032260a3a3c/volumes/kubernetes.io~projected/kube-api-access-4lrmh major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/471b293f-bc2d-4a9e-bddb-aea255ac235e/volumes/kubernetes.io~projected/kube-api-access-twrkf:{mountpoint:/var/lib/kubelet/pods/471b293f-bc2d-4a9e-bddb-aea255ac235e/volumes/kubernetes.io~projected/kube-api-access-twrkf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d24ab0e-8485-45c0-8b05-a149634357ff/volumes/kubernetes.io~projected/kube-api-access-vvzqm:{mountpoint:/var/lib/kubelet/pods/4d24ab0e-8485-45c0-8b05-a149634357ff/volumes/kubernetes.io~projected/kube-api-access-vvzqm major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d24ab0e-8485-45c0-8b05-a149634357ff/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/4d24ab0e-8485-45c0-8b05-a149634357ff/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58706aa4-9bad-4e69-abdb-b9af8390f27c/volumes/kubernetes.io~projected/kube-api-access-2jsmr:{mountpoint:/var/lib/kubelet/pods/58706aa4-9bad-4e69-abdb-b9af8390f27c/volumes/kubernetes.io~projected/kube-api-access-2jsmr major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58706aa4-9bad-4e69-abdb-b9af8390f27c/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/58706aa4-9bad-4e69-abdb-b9af8390f27c/volumes/kubernetes.io~secret/metrics-tls major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62ebb448-14fe-4919-9686-018754ede9cf/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/62ebb448-14fe-4919-9686-018754ede9cf/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62ebb448-14fe-4919-9686-018754ede9cf/volumes/kubernetes.io~projected/kube-api-access-jwzs8:{mountpoint:/var/lib/kubelet/pods/62ebb448-14fe-4919-9686-018754ede9cf/volumes/kubernetes.io~projected/kube-api-access-jwzs8 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62ebb448-14fe-4919-9686-018754ede9cf/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/62ebb448-14fe-4919-9686-018754ede9cf/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6db45ae1-4ba3-466f-82f9-005ce3d82bd2/volumes/kubernetes.io~projected/kube-api-access-jnbqs:{mountpoint:/var/lib/kubelet/pods/6db45ae1-4ba3-466f-82f9-005ce3d82bd2/volumes/kubernetes.io~projected/kube-api-access-jnbqs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d/volumes/kubernetes.io~projected/kube-api-access-t2b5z:{mountpoint:/var/lib/kubelet/pods/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d/volumes/kubernetes.io~projected/kube-api-access-t2b5z major:0 minor:119 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:118 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:114 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/73ae3560-963a-46a6-a052-21db48a98360/volumes/kubernetes.io~projected/kube-api-access-bnjc9:{mountpoint:/var/lib/kubelet/pods/73ae3560-963a-46a6-a052-21db48a98360/volumes/kubernetes.io~projected/kube-api-access-bnjc9 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91502bdf-1f01-469f-ac9f-64aa13e5563a/volumes/kubernetes.io~projected/kube-api-access-7sj9m:{mountpoint:/var/lib/kubelet/pods/91502bdf-1f01-469f-ac9f-64aa13e5563a/volumes/kubernetes.io~projected/kube-api-access-7sj9m major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93cbeaeb-857f-4c3f-825f-62c46320edd0/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/93cbeaeb-857f-4c3f-825f-62c46320edd0/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff/volumes/kubernetes.io~projected/kube-api-access-d97vg:{mountpoint:/var/lib/kubelet/pods/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff/volumes/kubernetes.io~projected/kube-api-access-d97vg major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384/volumes/kubernetes.io~projected/kube-api-access-dc4m9:{mountpoint:/var/lib/kubelet/pods/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384/volumes/kubernetes.io~projected/kube-api-access-dc4m9 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd9d674b-c3d9-4e78-bc55-81cedfc32881/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cd9d674b-c3d9-4e78-bc55-81cedfc32881/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e70c8827-8921-4231-8144-395f162320c2/volumes/kubernetes.io~projected/kube-api-access-n2gzd:{mountpoint:/var/lib/kubelet/pods/e70c8827-8921-4231-8144-395f162320c2/volumes/kubernetes.io~projected/kube-api-access-n2gzd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e70c8827-8921-4231-8144-395f162320c2/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/e70c8827-8921-4231-8144-395f162320c2/volumes/kubernetes.io~secret/metrics-certs major:0 minor:261 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d10e74ec7e59d9a8ff249bbf6fa5a5e1aa3100e8678a3c64e8ee175e6fd6034a/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f4963f982426b9f30f65960961a12d793a0054c0dbd74bbfbe204b3dd8781d87/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/ecf9393b7e83f3d4e458c336b5a27cdf70de6c24096ea5734029e8404d55c1d3/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-122:{mountpoint:/var/lib/containers/storage/overlay/29d712ee6ed2025ce0f99ecec32e26459deedb21bdcb9b79a80cb19fe347a2ff/merged major:0 minor:122 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/67724312a3e93c9dfe01cf7f6efac853a6e6200f3c21d81a23071958fbd6165a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-126:{mountpoint:/var/lib/containers/storage/overlay/faae678703af4a7e90fd4b65b113f5a4b26ccbc4b5c2f40048d6b799263ca865/merged major:0 minor:126 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/3deefe08329b5eb69bbe914ea351f9046457ada622af2ad6649466c9e0f117f3/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/00dd08a55683039463c8ce3eb030bf096efe239f2d9a52d32959b4106f862923/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/1d7cee28e80ed81fe8c926b89e56284e570f2b47c1b8fe53d58c2201d4543d88/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/6525362aa55f4f458a7fa6651e717409e0b76651032da2adfce09ef7b9227df3/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/6ff39f0075855e37658a1e86e3ae3801c24830e2a533a47f5be57e5bd7b675fd/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/41f4976bcb8c7078e371bd3fe7b207de60802d87a2f73f1aeff6ca779ae2b39a/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-156:{mountpoint:/var/lib/containers/storage/overlay/a791e0c61ab85a83b2f7f0ec1b242be66294b46493afef9ba8bbd91dcc879329/merged major:0 minor:156 fsType:overlay blockSize:0} overlay_0-165:{mountpoint:/var/lib/containers/storage/overlay/980f4a7eaef679c6eb713ee66ee34043d3286bd037fbd3877b570c79206e1dbd/merged major:0 minor:165 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/adda807ad0080aff8da03ba30b5e11d7edc7430e863971f7c65aa882241f0f67/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/52a9aa78f39935713dc871de217c155f7a743f3b37c7bff54df8782f5f334317/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-179:{mountpoint:/var/lib/containers/storage/overlay/4346f8cf2211d387f256ebc580951b0c68aa7aafe55060f69f23fd41fe7c1dac/merged major:0 minor:179 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/472046389e0c41983d8eb753a2a656d95c965528a9511e32cf55d7d670804783/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/93d1987c3cebc7d3df21f3f955789269174a08de953751a54622ddc61c0b056d/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-194:{mountpoint:/var/lib/containers/storage/overlay/7c15a55fc6373094aa3a9c2d0325b4dccecde41fca510bdbd5047e683262d4ac/merged major:0 minor:194 fsType:overlay blockSize:0} overlay_0-199:{mountpoint:/var/lib/containers/storage/overlay/227a156216b831d0f8be719df48e149e104aa444085ca3d84d5decaac85428e4/merged major:0 minor:199 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/07f058675e5e705b2873c6a8f1d9fcc3ec6f97aa10a29296b2b6ef6fb35f2fb4/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/94d8c21b84ac851b77a61642964cae06bc1772b6899cf4933a6cc03b827d41d0/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/1fd17eec7513e1bf06f9353373fae400969dd13b88286ba1e962ceb51c0656e5/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/3c3c4612a11164c506440d9317e7ed5f84a2d98389d24e9e7bbca9cb3283cbb6/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/0b898f2707327efbddcfb8f240b20253980d4a5e8377d7935257be0aab892d05/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/6163dc398da702c3037089dc59478dd66be404aa7af4060a00ef47f52f90e37d/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/f5fc224884af11bfb1d144adbc0c5e278e7ffbc5ed306f3531c462dd83c89062/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/6d3f87facfadfbd194998f02682aa8799b7be772c37edc8bd04584084ba5a69f/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/f91efe4a02426c3ee7a9a3d62825aacbca2bb25299a99ae16ee28a61e31d9eb2/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/c5a8231d978f5e6073513024358e122c7baebbb3f4048f85d5ec5f790173eacb/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/8b7cec1a2dd8437f8894f77dc28b9c41befe32a47da71e3925a9f917afabd27d/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/ded3b87b94fd3dce00e92b7ac7f1326da56b91eba172140aee8aab83198382dc/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/220b813b52d132739c68ff57ec9048e7a2006e8b1358c3b7231fd20924af4ffe/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/a4ffe067bb92727e8c1f9060b855d83ce8a098b61d651a503e775429957fc686/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/b62c62e2830dc31c546c86fb0c56d1bf71e77de46f4ee365ccf6efc64683047c/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/9778c9d8ce1b346cb73406f969cbb8c55d976cc172cc636f3e53a87edc4657da/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/f2e4a9579968e30101bf2f81ff9e5797087d97c37f4e36e37b3a641432b1b2f4/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/25ef1464b28532107841664cb17c26459a527b15647afad17755e1a673100e4a/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/091b106aaab3bf5f469d7dd282d0174f548870c2c1810d8aebbaeca952b9a41c/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ab147879f03d55a14b79b35a708117b9e57267b74ef3e90036e0c54fc3dc9bf7/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/235061bc4f1a9ac7b836dd5f868db6139433675f4f11e402a981655fad42496d/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d3b9f167218cb73b0164f4e9f4eb1e5b663dfe6257ad4ad5be8bf13e3071836c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c4e433c29cb91fe2863ee719da6d4626769e2be882daf5fa488f597f8e9edc92/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3d33f173e0e08d962176363b3af3cdb6d83f3837cbd70c51c6a44e3130d7dbf3/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/7f0690aa68ec81b6526a2dece1a52728c153de9a0f065a6d5b9e5fbd07e08bbb/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/8f6895eef0a0f345c20106cf39730b30db788bd62adc72a292588967272c8206/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/95bf811d18f5e37ff6272fb143f06abba30cbfc237ae90d029b6fbc044fae054/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/d370eb8309198d53de1dcb017b965d4c7cc5fb747454edf0c915991a33ee5ea7/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/ad60045675113880c2e4565194b69c9061cee0215f02c9818b205c3bee5330a9/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.786865 4793 manager.go:217] Machine: {Timestamp:2026-10-30 03:57:56.78640373 +0000 UTC m=+0.072278497 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102433 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f587eec2a204283236b8693754f2 SystemUUID:ec22f587-eec2-a204-2832-36b8693754f2 BootID:fbbe49e2-6be6-4aff-96b1-5679c120a83d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384/volumes/kubernetes.io~projected/kube-api-access-dc4m9 DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff/volumes/kubernetes.io~projected/kube-api-access-d97vg DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4943309beae3fa168b16fa6651ee168faea5dabc32e9fa322e8de13241bd72aa/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b323e055211f8138d7a3042ac59e64b439fc2fe9726d4403697c69548f827853/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-199 DeviceMajor:0 DeviceMinor:199 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6db45ae1-4ba3-466f-82f9-005ce3d82bd2/volumes/kubernetes.io~projected/kube-api-access-jnbqs DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-165 DeviceMajor:0 DeviceMinor:165 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d24ab0e-8485-45c0-8b05-a149634357ff/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:224 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3f37f9ad9ba315f552b7c4d2625b92b253a54e0cdb45de54ff509d158c4fa9ca/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/471b293f-bc2d-4a9e-bddb-aea255ac235e/volumes/kubernetes.io~projected/kube-api-access-twrkf DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-194 DeviceMajor:0 DeviceMinor:194 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62ebb448-14fe-4919-9686-018754ede9cf/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0fda4935-bef9-407b-9ef9-ea0725f519f2/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/62b0c2bb8e22a9ee06b1009118d960ea8359a4b723d4e8a349590afee949810d/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9e3a029cd0b30c569bc987d45c211a622d84cbe6a04cf31e8ec796f51f14273/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c0ddd85ce8284af9f2e2b438c86be40ea67764495e5ceaa1fab27d8513cf984/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aee539f33e518dc52781f4c17bd7d1a48147663fcd223b756f32a58cf8a17e7c/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44b8082218dde0674055a08592a46b04f75ca8da879155d82b9f460e35a83f6c/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-179 DeviceMajor:0 DeviceMinor:179 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e70c8827-8921-4231-8144-395f162320c2/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:261 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e70c8827-8921-4231-8144-395f162320c2/volumes/kubernetes.io~projected/kube-api-access-n2gzd DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1c471f71-4a76-4003-9f66-663b894892c0/volumes/kubernetes.io~projected/kube-api-access-bns9g DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:114 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d/volumes/kubernetes.io~projected/kube-api-access-t2b5z DeviceMajor:0 DeviceMinor:119 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd9d674b-c3d9-4e78-bc55-81cedfc32881/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-122 DeviceMajor:0 DeviceMinor:122 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58706aa4-9bad-4e69-abdb-b9af8390f27c/volumes/kubernetes.io~projected/kube-api-access-2jsmr DeviceMajor:0 DeviceMinor:219 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae9e5f64487b9cf647c340cd997cbd4053cccf6cd4c3ad1436f51171cd28a9cf/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e2288c6b2aff49155d3a73d26b5ec61a3c4ea316a699466006c19c68dea61d6b/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62ebb448-14fe-4919-9686-018754ede9cf/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/438677a3-904a-474e-bbed-4032260a3a3c/volumes/kubernetes.io~projected/kube-api-access-4lrmh DeviceMajor:0 DeviceMinor:269 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58706aa4-9bad-4e69-abdb-b9af8390f27c/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:214 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abbab493d5f0878d5c4b363947ca2b72eaa617147509ef013c6f276f07121baa/userdata/shm DeviceMajor:0 DeviceMinor:274 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cb0599709a1b704df4591cc726efdc7ef213d151d80052bcc8727d3832aa53e4/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:118 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/91502bdf-1f01-469f-ac9f-64aa13e5563a/volumes/kubernetes.io~projected/kube-api-access-7sj9m DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fda4935-bef9-407b-9ef9-ea0725f519f2/volumes/kubernetes.io~projected/kube-api-access-z72tk DeviceMajor:0 DeviceMinor:218 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b53ba5682503e82dce02a1cf8ab9edb1e81f2ca6125a5f2e139764f496e5bd14/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/73ae3560-963a-46a6-a052-21db48a98360/volumes/kubernetes.io~projected/kube-api-access-bnjc9 DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/004d1a87951da880edfce6abcf22b6583af05d032b6b9e1af79ea47b03f5fc85/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1befeb0a3b86351b7c14a4e2e2d45c17eecbd5b0b65949d1d0be4616abcc6fb/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62ebb448-14fe-4919-9686-018754ede9cf/volumes/kubernetes.io~projected/kube-api-access-jwzs8 DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae5cb5e8f5b97a78add2edabad7c2e76ae5fa5c3fe25ef9df4910ca1bd2045d3/userdata/shm DeviceMajor:0 DeviceMinor:120 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-126 DeviceMajor:0 DeviceMinor:126 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-156 DeviceMajor:0 DeviceMinor:156 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fe46aaf172fc3c4e140fde06e9f5a11e5f35ae15d821a71324c1db5589b9ed3/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4d24ab0e-8485-45c0-8b05-a149634357ff/volumes/kubernetes.io~projected/kube-api-access-vvzqm DeviceMajor:0 DeviceMinor:225 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/93cbeaeb-857f-4c3f-825f-62c46320edd0/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:260 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3f37f9ad9ba315f MacAddress:1a:a1:95:ac:f1:54 Speed:10000 Mtu:8901} {Name:4fe46aaf172fc3c MacAddress:92:44:05:5e:b4:57 Speed:10000 Mtu:8901} {Name:abbab493d5f0878 MacAddress:e6:7a:fa:8f:64:0c Speed:10000 Mtu:8901} {Name:ae9e5f64487b9cf MacAddress:7e:3d:e1:ec:d8:0d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c2:ed:81:29 Speed:0 Mtu:9001} {Name:br-int MacAddress:92:0a:f1:af:c0:d5 Speed:0 Mtu:8901} {Name:cb0599709a1b704 MacAddress:42:94:45:c0:94:e9 Speed:10000 Mtu:8901} {Name:e2288c6b2aff491 MacAddress:2e:83:c2:60:4e:62 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c2:ed:81:29 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:de:7b:0f:59:49:ed Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6e:8e:c4:ab:8f:65 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787128 4793 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787166 4793 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787436 4793 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787454 4793 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-177.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787572 4793 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787580 4793 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787593 4793 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787606 4793 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787686 4793 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787823 4793 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787923 4793 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787963 4793 kubelet.go:493] "Attempting to sync node with API server" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787975 4793 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787985 4793 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.787994 4793 kubelet.go:405] "Adding apiserver pod source" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.788002 4793 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.788709 4793 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.788722 4793 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 03:57:56.792096 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.789176 4793 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 03:57:56.793259 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.793133 4793 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 03:57:56.793259 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.793145 4793 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 03:57:56.793368 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.793265 4793 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 03:57:56.793928 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.793911 4793 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 03:57:56.794007 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.793952 4793 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 03:57:56.794099 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794086 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 03:57:56.794150 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794112 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 03:57:56.794150 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794123 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 03:57:56.794150 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794132 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 03:57:56.794150 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794141 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 03:57:56.794150 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794151 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 03:57:56.794380 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794161 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 03:57:56.794380 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794171 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 03:57:56.794380 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794183 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 03:57:56.794380 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794192 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 03:57:56.794380 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794212 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 03:57:56.794380 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794227 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 03:57:56.794380 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794294 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 03:57:56.794380 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.794304 4793 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 03:57:56.797948 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.797776 4793 server.go:1263] "Started kubelet" Oct 30 03:57:56.798504 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.797853 4793 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 03:57:56.798675 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.798481 4793 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 03:57:56.798675 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.798589 4793 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 03:57:56.798585 ip-10-0-138-177 systemd[1]: Started Kubernetes Kubelet. Oct 30 03:57:56.799619 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.799573 4793 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 03:57:56.804444 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.799862 4793 server.go:317] "Adding debug handlers to kubelet server" Oct 30 03:57:56.805510 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.805493 4793 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 03:57:56.805975 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.805955 4793 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 03:57:56.805975 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.805975 4793 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 03:57:56.806119 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.806034 4793 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 03:57:56.806167 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.806119 4793 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 03:57:56.806384 ip-10-0-138-177 kubenswrapper[4793]: E1030 03:57:56.806345 4793 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-177.ec2.internal\" not found" Oct 30 03:57:56.806494 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.806347 4793 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 03:52:18 +0000 UTC" deadline="2028-06-24 19:54:24.529305319 +0000 UTC" Oct 30 03:57:56.806494 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.806395 4793 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14487h56m27.72291452s" Oct 30 03:57:56.819545 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.819525 4793 factory.go:55] Registering systemd factory Oct 30 03:57:56.819545 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.819547 4793 factory.go:223] Registration of the systemd container factory successfully Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.819835 4793 factory.go:153] Registering CRI-O factory Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.819861 4793 factory.go:223] Registration of the crio container factory successfully Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.819922 4793 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.819944 4793 factory.go:103] Registering Raw factory Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.819961 4793 manager.go:1199] Started watching for new ooms in manager Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.819942 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471b293f-bc2d-4a9e-bddb-aea255ac235e" volumeName="kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-cni-binary-copy" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.819990 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d24ab0e-8485-45c0-8b05-a149634357ff" volumeName="kubernetes.io/configmap/4d24ab0e-8485-45c0-8b05-a149634357ff-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820006 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6db45ae1-4ba3-466f-82f9-005ce3d82bd2" volumeName="kubernetes.io/projected/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-kube-api-access-jnbqs" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820023 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="703f4255-11e5-4a34-a7d2-c1e26ef1fe3d" volumeName="kubernetes.io/configmap/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820038 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93cbeaeb-857f-4c3f-825f-62c46320edd0" volumeName="kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820051 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd9d674b-c3d9-4e78-bc55-81cedfc32881" volumeName="kubernetes.io/configmap/cd9d674b-c3d9-4e78-bc55-81cedfc32881-konnectivity-ca" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820067 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471b293f-bc2d-4a9e-bddb-aea255ac235e" volumeName="kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820082 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58706aa4-9bad-4e69-abdb-b9af8390f27c" volumeName="kubernetes.io/configmap/58706aa4-9bad-4e69-abdb-b9af8390f27c-config-volume" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820104 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62ebb448-14fe-4919-9686-018754ede9cf" volumeName="kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-env-overrides" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820118 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62ebb448-14fe-4919-9686-018754ede9cf" volumeName="kubernetes.io/projected/62ebb448-14fe-4919-9686-018754ede9cf-kube-api-access-jwzs8" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820133 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd9d674b-c3d9-4e78-bc55-81cedfc32881" volumeName="kubernetes.io/secret/cd9d674b-c3d9-4e78-bc55-81cedfc32881-agent-certs" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820148 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d24ab0e-8485-45c0-8b05-a149634357ff" volumeName="kubernetes.io/secret/4d24ab0e-8485-45c0-8b05-a149634357ff-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820162 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58706aa4-9bad-4e69-abdb-b9af8390f27c" volumeName="kubernetes.io/secret/58706aa4-9bad-4e69-abdb-b9af8390f27c-metrics-tls" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820174 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ae3560-963a-46a6-a052-21db48a98360" volumeName="kubernetes.io/configmap/73ae3560-963a-46a6-a052-21db48a98360-serviceca" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820189 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91502bdf-1f01-469f-ac9f-64aa13e5563a" volumeName="kubernetes.io/configmap/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-daemon-config" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820200 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9baf7ee6-a95f-42bc-8ddf-a197fb3737ff" volumeName="kubernetes.io/empty-dir/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-tmp" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820211 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9baf7ee6-a95f-42bc-8ddf-a197fb3737ff" volumeName="kubernetes.io/projected/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-kube-api-access-d97vg" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820226 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fda4935-bef9-407b-9ef9-ea0725f519f2" volumeName="kubernetes.io/secret/0fda4935-bef9-407b-9ef9-ea0725f519f2-cert" seLinuxMountContext="" Oct 30 03:57:56.820222 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820239 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d24ab0e-8485-45c0-8b05-a149634357ff" volumeName="kubernetes.io/projected/4d24ab0e-8485-45c0-8b05-a149634357ff-kube-api-access-vvzqm" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820250 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6db45ae1-4ba3-466f-82f9-005ce3d82bd2" volumeName="kubernetes.io/empty-dir/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-tmp-dir" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820262 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="73ae3560-963a-46a6-a052-21db48a98360" volumeName="kubernetes.io/projected/73ae3560-963a-46a6-a052-21db48a98360-kube-api-access-bnjc9" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820284 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5c3b257-dd0c-4af8-aa03-3efb4fb6a384" volumeName="kubernetes.io/projected/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-kube-api-access-dc4m9" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820296 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="703f4255-11e5-4a34-a7d2-c1e26ef1fe3d" volumeName="kubernetes.io/secret/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-tls" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820307 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58706aa4-9bad-4e69-abdb-b9af8390f27c" volumeName="kubernetes.io/projected/58706aa4-9bad-4e69-abdb-b9af8390f27c-kube-api-access-2jsmr" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820319 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62ebb448-14fe-4919-9686-018754ede9cf" volumeName="kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-ovnkube-script-lib" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820331 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62ebb448-14fe-4919-9686-018754ede9cf" volumeName="kubernetes.io/secret/62ebb448-14fe-4919-9686-018754ede9cf-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820338 4793 manager.go:319] Starting recovery of all containers Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820343 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91502bdf-1f01-469f-ac9f-64aa13e5563a" volumeName="kubernetes.io/configmap/91502bdf-1f01-469f-ac9f-64aa13e5563a-cni-binary-copy" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820355 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5c3b257-dd0c-4af8-aa03-3efb4fb6a384" volumeName="kubernetes.io/configmap/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-iptables-alerter-script" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820370 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e70c8827-8921-4231-8144-395f162320c2" volumeName="kubernetes.io/projected/e70c8827-8921-4231-8144-395f162320c2-kube-api-access-n2gzd" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820383 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fda4935-bef9-407b-9ef9-ea0725f519f2" volumeName="kubernetes.io/projected/0fda4935-bef9-407b-9ef9-ea0725f519f2-kube-api-access-z72tk" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820394 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471b293f-bc2d-4a9e-bddb-aea255ac235e" volumeName="kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820407 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d24ab0e-8485-45c0-8b05-a149634357ff" volumeName="kubernetes.io/empty-dir/4d24ab0e-8485-45c0-8b05-a149634357ff-data-volume" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820448 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="703f4255-11e5-4a34-a7d2-c1e26ef1fe3d" volumeName="kubernetes.io/configmap/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-metrics-client-ca" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820461 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9baf7ee6-a95f-42bc-8ddf-a197fb3737ff" volumeName="kubernetes.io/empty-dir/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-tuned" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820472 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="438677a3-904a-474e-bbed-4032260a3a3c" volumeName="kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820484 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="471b293f-bc2d-4a9e-bddb-aea255ac235e" volumeName="kubernetes.io/projected/471b293f-bc2d-4a9e-bddb-aea255ac235e-kube-api-access-twrkf" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820495 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62ebb448-14fe-4919-9686-018754ede9cf" volumeName="kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-ovnkube-config" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820506 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="703f4255-11e5-4a34-a7d2-c1e26ef1fe3d" volumeName="kubernetes.io/secret/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820518 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91502bdf-1f01-469f-ac9f-64aa13e5563a" volumeName="kubernetes.io/projected/91502bdf-1f01-469f-ac9f-64aa13e5563a-kube-api-access-7sj9m" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820529 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e70c8827-8921-4231-8144-395f162320c2" volumeName="kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820541 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58706aa4-9bad-4e69-abdb-b9af8390f27c" volumeName="kubernetes.io/empty-dir/58706aa4-9bad-4e69-abdb-b9af8390f27c-tmp-dir" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820554 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="703f4255-11e5-4a34-a7d2-c1e26ef1fe3d" volumeName="kubernetes.io/empty-dir/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-textfile" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820567 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="703f4255-11e5-4a34-a7d2-c1e26ef1fe3d" volumeName="kubernetes.io/projected/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-kube-api-access-t2b5z" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820580 4793 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c471f71-4a76-4003-9f66-663b894892c0" volumeName="kubernetes.io/projected/1c471f71-4a76-4003-9f66-663b894892c0-kube-api-access-bns9g" seLinuxMountContext="" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820591 4793 reconstruct.go:97] "Volume reconstruction finished" Oct 30 03:57:56.821470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.820598 4793 reconciler.go:29] "Reconciler: start to sync state" Oct 30 03:57:56.836188 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.836151 4793 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 03:57:56.837508 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.837485 4793 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 03:57:56.837592 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.837513 4793 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 03:57:56.837592 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.837530 4793 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 03:57:56.837592 ip-10-0-138-177 kubenswrapper[4793]: E1030 03:57:56.837563 4793 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 03:57:56.847195 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.847173 4793 generic.go:356] "Generic (PLEG): container finished" podID="471b293f-bc2d-4a9e-bddb-aea255ac235e" containerID="6cb6adf2935a3db4a727d643af264bbbacbf85516251e5b8a4248b9ef6cb5ada" exitCode=0 Oct 30 03:57:56.847275 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.847197 4793 generic.go:356] "Generic (PLEG): container finished" podID="471b293f-bc2d-4a9e-bddb-aea255ac235e" containerID="29483355a2f69cb0135447c5a6436782f670670e09fc35df11bd2a71548d40c4" exitCode=0 Oct 30 03:57:56.847275 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.847204 4793 generic.go:356] "Generic (PLEG): container finished" podID="471b293f-bc2d-4a9e-bddb-aea255ac235e" containerID="65aad0390297bd80d1f1d3e14abeffc0ec823691dc4543d0ed79cb93aa22f42e" exitCode=0 Oct 30 03:57:56.847275 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.847210 4793 generic.go:356] "Generic (PLEG): container finished" podID="471b293f-bc2d-4a9e-bddb-aea255ac235e" containerID="dbe1b27ab3dc1c8f833f3ee9600a1460a3a61d7937b175d4b2e9e4bb9b2faa2d" exitCode=0 Oct 30 03:57:56.847275 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.847215 4793 generic.go:356] "Generic (PLEG): container finished" podID="471b293f-bc2d-4a9e-bddb-aea255ac235e" containerID="0bc5890876fabbc48ce7d8ebb357116880d35c0d394917910289d79a9bba3883" exitCode=0 Oct 30 03:57:56.847275 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.847220 4793 generic.go:356] "Generic (PLEG): container finished" podID="471b293f-bc2d-4a9e-bddb-aea255ac235e" containerID="219c8215555d34603f9393a89a39e4019dc6ca2fe649c10a6a9dc28f3ae2bec2" exitCode=0 Oct 30 03:57:56.852496 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.852476 4793 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:57:56.852650 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.852633 4793 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 03:57:56.853312 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.853295 4793 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:57:56.871334 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.871308 4793 generic.go:356] "Generic (PLEG): container finished" podID="703f4255-11e5-4a34-a7d2-c1e26ef1fe3d" containerID="c6997d00e1f3e286e39cdf6b42c1af4c760dd15811b86c4deb6338dc2236ebfd" exitCode=0 Oct 30 03:57:56.874734 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.874714 4793 generic.go:356] "Generic (PLEG): container finished" podID="3a1a550d76a223b097bf2829db01e97d" containerID="8dea47e83590faec28e80a182a13f3a67685e6d1a90a928b80c184d4e7171e1a" exitCode=0 Oct 30 03:57:56.877555 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.877536 4793 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:57:56.877786 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.877760 4793 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 03:57:56.897485 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.897468 4793 manager.go:324] Recovery completed Oct 30 03:57:56.913034 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.912992 4793 cpu_manager.go:226] "Starting" policy="none" Oct 30 03:57:56.913034 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.913005 4793 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 03:57:56.913034 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.913021 4793 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 03:57:56.913184 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.913134 4793 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 03:57:56.913184 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.913145 4793 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 03:57:56.913184 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.913161 4793 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 03:57:56.913184 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.913166 4793 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 03:57:56.913184 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.913171 4793 policy_none.go:50] "Start" Oct 30 03:57:56.913184 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.913182 4793 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 03:57:56.913347 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.913191 4793 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 03:57:56.918269 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.918255 4793 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 03:57:56.918305 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.918270 4793 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 03:57:56.918305 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.918276 4793 policy_none.go:44] "Start" Oct 30 03:57:56.922192 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.922168 4793 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 03:57:56.924788 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.924776 4793 manager.go:341] "Starting Device Plugin manager" Oct 30 03:57:56.925277 ip-10-0-138-177 kubenswrapper[4793]: E1030 03:57:56.924800 4793 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 03:57:56.925353 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.925280 4793 server.go:87] "Starting device plugin registration server" Oct 30 03:57:56.925538 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.925526 4793 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 03:57:56.925608 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.925537 4793 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 03:57:56.925664 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.925640 4793 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 03:57:56.925776 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.925734 4793 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 03:57:56.925895 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.925782 4793 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 03:57:56.925895 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.925789 4793 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 03:57:56.925996 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.925827 4793 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T03:57:56.92577041Z","UUID":"ff518db9-ec76-40e6-acab-985f097436d0","Handler":null,"Name":"","Endpoint":""} Oct 30 03:57:56.927443 ip-10-0-138-177 kubenswrapper[4793]: E1030 03:57:56.927401 4793 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 03:57:56.927873 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.927847 4793 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 03:57:56.927873 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.927873 4793 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 03:57:56.937664 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.937634 4793 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal"] Oct 30 03:57:56.938246 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.938216 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" event={"ID":"552cd6bdf9ef61710ec1ae6f1eae6b32","Type":"ContainerStarted","Data":"1fe70b0f577fc3d5cb4af58dc38cbee39a911c41090b483da763b153fd5ee3d9"} Oct 30 03:57:56.938246 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.938238 4793 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:56.938413 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.938255 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" event={"ID":"552cd6bdf9ef61710ec1ae6f1eae6b32","Type":"ContainerStarted","Data":"b53ba5682503e82dce02a1cf8ab9edb1e81f2ca6125a5f2e139764f496e5bd14"} Oct 30 03:57:56.938413 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.938294 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" event={"ID":"3a1a550d76a223b097bf2829db01e97d","Type":"ContainerStarted","Data":"0e9b6b5d462ce55533fa5ebc9d065585ad252f3f87a3ff6a1fc81716610bad79"} Oct 30 03:57:56.938413 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.938314 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" event={"ID":"3a1a550d76a223b097bf2829db01e97d","Type":"ContainerDied","Data":"8dea47e83590faec28e80a182a13f3a67685e6d1a90a928b80c184d4e7171e1a"} Oct 30 03:57:56.938413 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.938332 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" event={"ID":"3a1a550d76a223b097bf2829db01e97d","Type":"ContainerStarted","Data":"2c0ddd85ce8284af9f2e2b438c86be40ea67764495e5ceaa1fab27d8513cf984"} Oct 30 03:57:56.938628 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.938495 4793 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:56.964956 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.964936 4793 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:57:56.965035 ip-10-0-138-177 kubenswrapper[4793]: E1030 03:57:56.964977 4793 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-177.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:56.965431 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:56.965401 4793 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:57:56.965473 ip-10-0-138-177 kubenswrapper[4793]: E1030 03:57:56.965455 4793 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.025630 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.025608 4793 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 03:57:57.026620 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.026606 4793 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientMemory" Oct 30 03:57:57.026700 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.026630 4793 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 03:57:57.026700 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.026639 4793 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-177.ec2.internal" event="NodeHasSufficientPID" Oct 30 03:57:57.026700 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.026670 4793 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.054130 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.054108 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/552cd6bdf9ef61710ec1ae6f1eae6b32-config\") pod \"kube-apiserver-proxy-ip-10-0-138-177.ec2.internal\" (UID: \"552cd6bdf9ef61710ec1ae6f1eae6b32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.054211 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.054133 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a1a550d76a223b097bf2829db01e97d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" (UID: \"3a1a550d76a223b097bf2829db01e97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.054211 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.054151 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a1a550d76a223b097bf2829db01e97d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" (UID: \"3a1a550d76a223b097bf2829db01e97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.061518 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.061502 4793 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.061593 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.061546 4793 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.154815 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.154794 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/552cd6bdf9ef61710ec1ae6f1eae6b32-config\") pod \"kube-apiserver-proxy-ip-10-0-138-177.ec2.internal\" (UID: \"552cd6bdf9ef61710ec1ae6f1eae6b32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.154912 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.154817 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a1a550d76a223b097bf2829db01e97d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" (UID: \"3a1a550d76a223b097bf2829db01e97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.154912 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.154834 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a1a550d76a223b097bf2829db01e97d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" (UID: \"3a1a550d76a223b097bf2829db01e97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.154912 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.154878 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a1a550d76a223b097bf2829db01e97d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" (UID: \"3a1a550d76a223b097bf2829db01e97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.155031 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.154908 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/552cd6bdf9ef61710ec1ae6f1eae6b32-config\") pod \"kube-apiserver-proxy-ip-10-0-138-177.ec2.internal\" (UID: \"552cd6bdf9ef61710ec1ae6f1eae6b32\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.155031 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.154926 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3a1a550d76a223b097bf2829db01e97d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" (UID: \"3a1a550d76a223b097bf2829db01e97d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.789022 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.788991 4793 apiserver.go:52] "Watching apiserver" Oct 30 03:57:57.830519 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.830497 4793 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 03:57:57.830909 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.830890 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-lpm6b","kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal","openshift-ingress-canary/ingress-canary-wq96k","openshift-dns/node-resolver-zjzzn","openshift-network-operator/iptables-alerter-q98qf","kube-system/konnectivity-agent-9gghj","openshift-insights/insights-runtime-extractor-b7bml","openshift-multus/multus-9fmv2","kube-system/global-pull-secret-syncer-p2mmh","openshift-cluster-node-tuning-operator/tuned-dmdgx","openshift-monitoring/node-exporter-d8rtc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x","openshift-dns/dns-default-zrv7n","openshift-multus/network-metrics-daemon-tffth","openshift-ovn-kubernetes/ovnkube-node-krggs","openshift-image-registry/node-ca-x58t9","openshift-network-diagnostics/network-check-target-v5zp7"] Oct 30 03:57:57.846692 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.846665 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-vnctk\"" Oct 30 03:57:57.846692 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.846687 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-smvt7\"" Oct 30 03:57:57.846841 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.846670 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 03:57:57.846841 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.846747 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j9mnz\"" Oct 30 03:57:57.846979 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.846957 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 03:57:57.847051 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.847009 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 03:57:57.847051 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.847016 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 03:57:57.847051 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.847025 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 03:57:57.847051 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.847031 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 03:57:57.847184 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.847054 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 03:57:57.849556 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.849542 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 03:57:57.849631 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.849616 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vm8lh\"" Oct 30 03:57:57.860121 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.860099 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 03:57:57.860207 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.860127 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 03:57:57.860516 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.860446 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-dnvhh\"" Oct 30 03:57:57.860759 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.860736 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rs54w\"" Oct 30 03:57:57.860844 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.860778 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:57:57.860844 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.860791 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h7p9v\"" Oct 30 03:57:57.861067 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861032 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8s8mj\"" Oct 30 03:57:57.861067 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861045 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 03:57:57.861067 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861066 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 03:57:57.861472 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861455 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 03:57:57.861472 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861468 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 03:57:57.861591 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861459 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 03:57:57.861591 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861489 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mwxht\"" Oct 30 03:57:57.861591 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861473 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 03:57:57.861591 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861460 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 03:57:57.861591 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861515 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 03:57:57.861591 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861538 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 03:57:57.861591 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861546 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 03:57:57.861591 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861569 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 03:57:57.861591 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861573 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 03:57:57.861591 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861570 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861599 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861615 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kwf82\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861661 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861692 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-skmdf\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861725 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861736 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861755 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b2qtm\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861765 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861769 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861782 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-846td\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861786 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861795 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861810 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861818 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 03:57:57.862077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.861822 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 03:57:57.862621 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862602 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 03:57:57.862621 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862605 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 03:57:57.862771 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862641 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 03:57:57.862771 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862648 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 03:57:57.862771 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862651 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4xznk\"" Oct 30 03:57:57.862771 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862661 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w4blj\"" Oct 30 03:57:57.862771 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862643 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hmk7j\"" Oct 30 03:57:57.862771 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862676 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 03:57:57.862771 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862679 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 03:57:57.862771 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862711 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 03:57:57.862771 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862707 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 03:57:57.862771 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862719 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 03:57:57.863036 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862845 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 03:57:57.863036 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.862893 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 03:57:57.884057 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.884037 4793 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.884677 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.884660 4793 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.907081 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.907058 4793 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 03:57:57.907507 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.907492 4793 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:57:57.907567 ip-10-0-138-177 kubenswrapper[4793]: E1030 03:57:57.907536 4793 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-177.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.908063 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.908051 4793 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 03:57:57.908111 ip-10-0-138-177 kubenswrapper[4793]: E1030 03:57:57.908081 4793 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-177.ec2.internal" Oct 30 03:57:57.958670 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958646 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.958774 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958679 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-etc-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.958774 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958744 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-metrics-client-ca\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.958774 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958767 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d24ab0e-8485-45c0-8b05-a149634357ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:57.958974 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958790 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-socket-dir-parent\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.958974 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958818 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-tmp-dir\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:57.958974 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958843 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-cni-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.958974 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958866 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-k8s-cni-cncf-io\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.958974 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958889 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysctl-conf\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.958974 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958914 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrkf\" (UniqueName: \"kubernetes.io/projected/471b293f-bc2d-4a9e-bddb-aea255ac235e-kube-api-access-twrkf\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:57.958974 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958944 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58706aa4-9bad-4e69-abdb-b9af8390f27c-config-volume\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:57.958974 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958968 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-slash\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.958995 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-run-netns\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959015 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4d24ab0e-8485-45c0-8b05-a149634357ff-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959017 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-node-log\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959065 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-wtmp\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959092 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d24ab0e-8485-45c0-8b05-a149634357ff-data-volume\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959153 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-cni-netd\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959175 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4d24ab0e-8485-45c0-8b05-a149634357ff-data-volume\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959181 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-sys-fs\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959207 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-textfile\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959232 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-modprobe-d\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959254 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-run\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959253 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-tmp-dir\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959154 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-metrics-client-ca\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.959338 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959289 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-host\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.959902 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959344 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58706aa4-9bad-4e69-abdb-b9af8390f27c-config-volume\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:57.959902 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959371 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-textfile\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.959902 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959445 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-tuned\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.959902 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959529 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-env-overrides\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.959902 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959532 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-tuned\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.959902 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959571 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvzqm\" (UniqueName: \"kubernetes.io/projected/4d24ab0e-8485-45c0-8b05-a149634357ff-kube-api-access-vvzqm\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:57.959902 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959609 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91502bdf-1f01-469f-ac9f-64aa13e5563a-cni-binary-copy\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.959902 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959653 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-cni-multus\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.959902 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959677 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-hostroot\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.959902 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.959722 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-daemon-config\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.960292 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960031 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-etc-kubernetes\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.960292 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960134 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91502bdf-1f01-469f-ac9f-64aa13e5563a-cni-binary-copy\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.960292 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960155 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-env-overrides\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.960292 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960161 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrmh\" (UniqueName: \"kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh\") pod \"network-check-target-v5zp7\" (UID: \"438677a3-904a-474e-bbed-4032260a3a3c\") " pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:57.960292 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960225 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-daemon-config\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.960547 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960304 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-device-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:57.960547 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960343 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z72tk\" (UniqueName: \"kubernetes.io/projected/0fda4935-bef9-407b-9ef9-ea0725f519f2-kube-api-access-z72tk\") pod \"ingress-canary-wq96k\" (UID: \"0fda4935-bef9-407b-9ef9-ea0725f519f2\") " pod="openshift-ingress-canary/ingress-canary-wq96k" Oct 30 03:57:57.960547 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960476 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-multus-certs\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.960547 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960518 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjc9\" (UniqueName: \"kubernetes.io/projected/73ae3560-963a-46a6-a052-21db48a98360-kube-api-access-bnjc9\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:57.960735 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960645 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-sys\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.960735 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960692 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-tmp\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.960735 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960717 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-system-cni-dir\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:57.960826 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960747 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-sys\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.960826 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960771 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-tmp\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.960826 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960774 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-kubernetes\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.960826 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960820 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d97vg\" (UniqueName: \"kubernetes.io/projected/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-kube-api-access-d97vg\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.960965 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960847 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-socket-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:57.960965 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960885 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-registration-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:57.960965 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960918 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-tls\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.960965 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960942 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-cni-bin\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.961077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.960969 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62ebb448-14fe-4919-9686-018754ede9cf-ovn-node-metrics-cert\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.961077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961045 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-kubelet\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.961077 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961068 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-netns\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.961162 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961093 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-ovnkube-config\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.961162 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961116 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:57.961162 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961139 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58706aa4-9bad-4e69-abdb-b9af8390f27c-tmp-dir\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:57.961251 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961165 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jsmr\" (UniqueName: \"kubernetes.io/projected/58706aa4-9bad-4e69-abdb-b9af8390f27c-kube-api-access-2jsmr\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:57.961251 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961190 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93cbeaeb-857f-4c3f-825f-62c46320edd0-kubelet-config\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:57.961304 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961271 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd9d674b-c3d9-4e78-bc55-81cedfc32881-konnectivity-ca\") pod \"konnectivity-agent-9gghj\" (UID: \"cd9d674b-c3d9-4e78-bc55-81cedfc32881\") " pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:57.961304 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961296 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-systemd\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.961356 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961329 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-ovn\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.961390 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961353 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-hosts-file\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:57.961436 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961384 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2b5z\" (UniqueName: \"kubernetes.io/projected/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-kube-api-access-t2b5z\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.961470 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961412 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-cni-bin\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.961511 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961468 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-kubelet\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.961547 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961526 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-conf-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.961579 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961555 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sj9m\" (UniqueName: \"kubernetes.io/projected/91502bdf-1f01-469f-ac9f-64aa13e5563a-kube-api-access-7sj9m\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.961608 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961580 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-lib-modules\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.961608 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961603 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-var-lib-kubelet\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.961660 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961631 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-var-lib-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.961691 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961671 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62ebb448-14fe-4919-9686-018754ede9cf-ovn-node-metrics-cert\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.961751 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961737 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.961781 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961765 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.961824 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961789 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d24ab0e-8485-45c0-8b05-a149634357ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:57.961824 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961815 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysconfig\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.961881 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961839 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysctl-d\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.961881 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961864 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-cnibin\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.961955 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961888 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:57.961955 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961914 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:57.961955 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961939 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-iptables-alerter-script\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:57.962038 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961966 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4m9\" (UniqueName: \"kubernetes.io/projected/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-kube-api-access-dc4m9\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:57.962038 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.961991 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bns9g\" (UniqueName: \"kubernetes.io/projected/1c471f71-4a76-4003-9f66-663b894892c0-kube-api-access-bns9g\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:57.962038 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962014 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fda4935-bef9-407b-9ef9-ea0725f519f2-cert\") pod \"ingress-canary-wq96k\" (UID: \"0fda4935-bef9-407b-9ef9-ea0725f519f2\") " pod="openshift-ingress-canary/ingress-canary-wq96k" Oct 30 03:57:57.962122 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962040 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73ae3560-963a-46a6-a052-21db48a98360-serviceca\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:57.962122 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962068 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:57.962122 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962093 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbqs\" (UniqueName: \"kubernetes.io/projected/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-kube-api-access-jnbqs\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:57.962122 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962117 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd9d674b-c3d9-4e78-bc55-81cedfc32881-agent-certs\") pod \"konnectivity-agent-9gghj\" (UID: \"cd9d674b-c3d9-4e78-bc55-81cedfc32881\") " pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:57.962226 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962142 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:57.962226 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962174 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-root\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.962226 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962198 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4d24ab0e-8485-45c0-8b05-a149634357ff-crio-socket\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:57.962309 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962257 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-system-cni-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.962309 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962288 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-ovnkube-config\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.962392 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962378 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-systemd\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.962438 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962404 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93cbeaeb-857f-4c3f-825f-62c46320edd0-dbus\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:57.962468 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962441 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-host-slash\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:57.962507 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962464 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73ae3560-963a-46a6-a052-21db48a98360-host\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:57.962507 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962492 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2gzd\" (UniqueName: \"kubernetes.io/projected/e70c8827-8921-4231-8144-395f162320c2-kube-api-access-n2gzd\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:57.962567 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962505 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fda4935-bef9-407b-9ef9-ea0725f519f2-cert\") pod \"ingress-canary-wq96k\" (UID: \"0fda4935-bef9-407b-9ef9-ea0725f519f2\") " pod="openshift-ingress-canary/ingress-canary-wq96k" Oct 30 03:57:57.962567 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962521 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwzs8\" (UniqueName: \"kubernetes.io/projected/62ebb448-14fe-4919-9686-018754ede9cf-kube-api-access-jwzs8\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.962567 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962546 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:57.962646 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962572 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-log-socket\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.962646 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962597 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.962646 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962624 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.962646 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962629 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.962747 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962654 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-ovnkube-script-lib\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.962747 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962674 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-cnibin\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:57.962747 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962694 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:57.962747 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962712 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58706aa4-9bad-4e69-abdb-b9af8390f27c-metrics-tls\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:57.962747 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962733 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:57.962881 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962749 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-systemd-units\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.962916 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.962881 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-accelerators-collector-config\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.963030 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.963017 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4d24ab0e-8485-45c0-8b05-a149634357ff-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:57.963216 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.963204 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:57.963249 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.963225 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-tls\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.963281 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.963250 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58706aa4-9bad-4e69-abdb-b9af8390f27c-tmp-dir\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:57.963614 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.963596 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-cni-binary-copy\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:57.963689 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.963675 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cd9d674b-c3d9-4e78-bc55-81cedfc32881-konnectivity-ca\") pod \"konnectivity-agent-9gghj\" (UID: \"cd9d674b-c3d9-4e78-bc55-81cedfc32881\") " pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:57.963823 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.963811 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/471b293f-bc2d-4a9e-bddb-aea255ac235e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:57.963889 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.963873 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62ebb448-14fe-4919-9686-018754ede9cf-ovnkube-script-lib\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:57.963966 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.963955 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58706aa4-9bad-4e69-abdb-b9af8390f27c-metrics-tls\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:57.964368 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.964352 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/93cbeaeb-857f-4c3f-825f-62c46320edd0-original-pull-secret\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:57.964400 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.964365 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73ae3560-963a-46a6-a052-21db48a98360-serviceca\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:57.964556 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.964543 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70c8827-8921-4231-8144-395f162320c2-metrics-certs\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:57.964709 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.964697 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-iptables-alerter-script\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:57.964860 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.964849 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cd9d674b-c3d9-4e78-bc55-81cedfc32881-agent-certs\") pod \"konnectivity-agent-9gghj\" (UID: \"cd9d674b-c3d9-4e78-bc55-81cedfc32881\") " pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:57:57.980920 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.980893 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvzqm\" (UniqueName: \"kubernetes.io/projected/4d24ab0e-8485-45c0-8b05-a149634357ff-kube-api-access-vvzqm\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:57.981013 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.980991 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twrkf\" (UniqueName: \"kubernetes.io/projected/471b293f-bc2d-4a9e-bddb-aea255ac235e-kube-api-access-twrkf\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:57.981089 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.981068 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrmh\" (UniqueName: \"kubernetes.io/projected/438677a3-904a-474e-bbed-4032260a3a3c-kube-api-access-4lrmh\") pod \"network-check-target-v5zp7\" (UID: \"438677a3-904a-474e-bbed-4032260a3a3c\") " pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:57:57.981584 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.981565 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjc9\" (UniqueName: \"kubernetes.io/projected/73ae3560-963a-46a6-a052-21db48a98360-kube-api-access-bnjc9\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:57.981948 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.981932 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z72tk\" (UniqueName: \"kubernetes.io/projected/0fda4935-bef9-407b-9ef9-ea0725f519f2-kube-api-access-z72tk\") pod \"ingress-canary-wq96k\" (UID: \"0fda4935-bef9-407b-9ef9-ea0725f519f2\") " pod="openshift-ingress-canary/ingress-canary-wq96k" Oct 30 03:57:57.983917 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.983901 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sj9m\" (UniqueName: \"kubernetes.io/projected/91502bdf-1f01-469f-ac9f-64aa13e5563a-kube-api-access-7sj9m\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:57.984189 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.984170 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jsmr\" (UniqueName: \"kubernetes.io/projected/58706aa4-9bad-4e69-abdb-b9af8390f27c-kube-api-access-2jsmr\") pod \"dns-default-zrv7n\" (UID: \"58706aa4-9bad-4e69-abdb-b9af8390f27c\") " pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:57.984230 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.984170 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d97vg\" (UniqueName: \"kubernetes.io/projected/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-kube-api-access-d97vg\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:57.986401 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.986381 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnbqs\" (UniqueName: \"kubernetes.io/projected/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-kube-api-access-jnbqs\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:57.986498 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.986389 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bns9g\" (UniqueName: \"kubernetes.io/projected/1c471f71-4a76-4003-9f66-663b894892c0-kube-api-access-bns9g\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:57.986778 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.986758 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2b5z\" (UniqueName: \"kubernetes.io/projected/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-kube-api-access-t2b5z\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:57.986831 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.986762 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4m9\" (UniqueName: \"kubernetes.io/projected/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-kube-api-access-dc4m9\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:57.987136 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.987119 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2gzd\" (UniqueName: \"kubernetes.io/projected/e70c8827-8921-4231-8144-395f162320c2-kube-api-access-n2gzd\") pod \"network-metrics-daemon-tffth\" (UID: \"e70c8827-8921-4231-8144-395f162320c2\") " pod="openshift-multus/network-metrics-daemon-tffth" Oct 30 03:57:57.987194 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:57.987142 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwzs8\" (UniqueName: \"kubernetes.io/projected/62ebb448-14fe-4919-9686-018754ede9cf-kube-api-access-jwzs8\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.063137 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063083 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-slash\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.063137 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063107 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-run-netns\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.063137 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063121 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-node-log\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.063137 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063135 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-wtmp\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063151 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-cni-netd\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063169 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-sys-fs\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063192 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-modprobe-d\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063202 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-slash\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063214 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-run\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063216 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-cni-netd\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063201 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-node-log\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063236 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-run-netns\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063250 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-sys-fs\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063260 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-run\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063248 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-host\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063292 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-host\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063294 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-cni-multus\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063324 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-cni-multus\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063323 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-hostroot\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063330 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-modprobe-d\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063334 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-node-exporter-wtmp\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063349 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-etc-kubernetes\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.063385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063354 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-hostroot\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063406 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-device-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063459 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-device-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063460 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-multus-certs\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063483 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-sys\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063493 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-multus-certs\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063408 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-etc-kubernetes\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063500 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-system-cni-dir\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063527 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-system-cni-dir\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063533 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-sys\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063541 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-sys\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063570 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-kubernetes\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063593 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-socket-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063602 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-sys\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063617 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-registration-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063641 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-kubernetes\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063642 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-cni-bin\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063659 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-socket-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063672 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-registration-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063671 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-kubelet\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063698 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-cni-bin\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063703 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-netns\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063711 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-kubelet\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063721 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93cbeaeb-857f-4c3f-825f-62c46320edd0-kubelet-config\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063741 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-systemd\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063754 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-ovn\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063774 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-hosts-file\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063792 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-cni-bin\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063792 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/93cbeaeb-857f-4c3f-825f-62c46320edd0-kubelet-config\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063789 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-systemd\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063815 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-netns\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063820 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-ovn\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063809 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-kubelet\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063837 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6db45ae1-4ba3-466f-82f9-005ce3d82bd2-hosts-file\") pod \"node-resolver-zjzzn\" (UID: \"6db45ae1-4ba3-466f-82f9-005ce3d82bd2\") " pod="openshift-dns/node-resolver-zjzzn" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063840 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-kubelet\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063834 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-var-lib-cni-bin\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063855 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-conf-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063874 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-lib-modules\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063896 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-var-lib-kubelet\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063897 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-conf-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063914 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-var-lib-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063942 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysconfig\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063954 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-lib-modules\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063960 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysctl-d\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063963 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-var-lib-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063958 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-var-lib-kubelet\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.063981 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-cnibin\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064009 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064029 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064030 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-cnibin\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064043 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-root\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064048 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysctl-d\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064072 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4d24ab0e-8485-45c0-8b05-a149634357ff-crio-socket\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064078 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/703f4255-11e5-4a34-a7d2-c1e26ef1fe3d-root\") pod \"node-exporter-d8rtc\" (UID: \"703f4255-11e5-4a34-a7d2-c1e26ef1fe3d\") " pod="openshift-monitoring/node-exporter-d8rtc" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064089 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-system-cni-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064088 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-etc-selinux\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064096 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064125 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-system-cni-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064125 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4d24ab0e-8485-45c0-8b05-a149634357ff-crio-socket\") pod \"insights-runtime-extractor-b7bml\" (UID: \"4d24ab0e-8485-45c0-8b05-a149634357ff\") " pod="openshift-insights/insights-runtime-extractor-b7bml" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064147 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-systemd\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064169 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93cbeaeb-857f-4c3f-825f-62c46320edd0-dbus\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064148 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysconfig\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.064161 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064201 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-systemd\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064204 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/93cbeaeb-857f-4c3f-825f-62c46320edd0-dbus\") pod \"global-pull-secret-syncer-p2mmh\" (UID: \"93cbeaeb-857f-4c3f-825f-62c46320edd0\") " pod="kube-system/global-pull-secret-syncer-p2mmh" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064227 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-host-slash\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064243 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73ae3560-963a-46a6-a052-21db48a98360-host\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064273 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73ae3560-963a-46a6-a052-21db48a98360-host\") pod \"node-ca-x58t9\" (UID: \"73ae3560-963a-46a6-a052-21db48a98360\") " pod="openshift-image-registry/node-ca-x58t9" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064273 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5c3b257-dd0c-4af8-aa03-3efb4fb6a384-host-slash\") pod \"iptables-alerter-q98qf\" (UID: \"a5c3b257-dd0c-4af8-aa03-3efb4fb6a384\") " pod="openshift-network-operator/iptables-alerter-q98qf" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064287 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064304 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-log-socket\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064311 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1c471f71-4a76-4003-9f66-663b894892c0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ll88x\" (UID: \"1c471f71-4a76-4003-9f66-663b894892c0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ll88x" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064319 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064337 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064354 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-cnibin\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064356 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064369 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-systemd-units\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064384 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064382 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064339 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-log-socket\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064411 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-systemd-units\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064440 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-run-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064440 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/471b293f-bc2d-4a9e-bddb-aea255ac235e-cnibin\") pod \"multus-additional-cni-plugins-lpm6b\" (UID: \"471b293f-bc2d-4a9e-bddb-aea255ac235e\") " pod="openshift-multus/multus-additional-cni-plugins-lpm6b" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064448 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-etc-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064473 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62ebb448-14fe-4919-9686-018754ede9cf-etc-openvswitch\") pod \"ovnkube-node-krggs\" (UID: \"62ebb448-14fe-4919-9686-018754ede9cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064498 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-socket-dir-parent\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064531 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-cni-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064556 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-k8s-cni-cncf-io\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064578 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysctl-conf\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064611 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-host-run-k8s-cni-cncf-io\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064615 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-cni-dir\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064639 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91502bdf-1f01-469f-ac9f-64aa13e5563a-multus-socket-dir-parent\") pod \"multus-9fmv2\" (UID: \"91502bdf-1f01-469f-ac9f-64aa13e5563a\") " pod="openshift-multus/multus-9fmv2" Oct 30 03:57:58.066008 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.064724 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9baf7ee6-a95f-42bc-8ddf-a197fb3737ff-etc-sysctl-conf\") pod \"tuned-dmdgx\" (UID: \"9baf7ee6-a95f-42bc-8ddf-a197fb3737ff\") " pod="openshift-cluster-node-tuning-operator/tuned-dmdgx" Oct 30 03:57:58.728983 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.728947 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:58.733316 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.730951 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zrv7n" Oct 30 03:57:58.935257 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:58.935220 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:57:59.150046 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:57:59.149966 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:03.186621 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:03.186588 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:03.992916 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:03.992876 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:05.321851 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:05.321824 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:58:05.703134 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:05.703112 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:58:06.373509 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:06.373481 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:58:06.373854 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:06.373671 4793 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 03:58:08.633537 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:08.633502 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:11.216723 ip-10-0-138-177 kubenswrapper[4793]: E1030 03:58:11.216687 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 03:58:11.927892 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:11.927863 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:12.207284 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:12.207197 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:12.430412 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:12.430376 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:20.338005 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.337970 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 03:58:20.340563 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.340545 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.357062 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.357037 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 03:58:20.357215 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.357038 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 03:58:20.357215 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.357084 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 03:58:20.370002 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.369976 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 03:58:20.370002 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.369992 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7cnb2\"" Oct 30 03:58:20.370002 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.370001 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 03:58:20.370154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.369997 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 03:58:20.370380 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.370363 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 03:58:20.382514 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.382493 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9c3566kc47c63\"" Oct 30 03:58:20.382616 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.382521 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 03:58:20.396304 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.396279 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 03:58:20.408114 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.408095 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 03:58:20.408213 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.408136 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 03:58:20.408213 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.408165 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 03:58:20.408213 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.408176 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 03:58:20.408213 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.408196 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 03:58:20.408368 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.408220 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 03:58:20.408368 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.408253 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 03:58:20.489499 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489477 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.489594 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489502 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.489594 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489524 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.489594 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489540 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-web-config\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.489759 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489660 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.489759 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489707 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.489759 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489746 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.489911 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489772 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.489911 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489826 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4075a738-d882-4257-9997-593dfc82570d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.489911 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489863 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4075a738-d882-4257-9997-593dfc82570d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.489911 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489900 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4075a738-d882-4257-9997-593dfc82570d-config-out\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.490072 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489927 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.490072 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489962 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.490072 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.489988 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.490072 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.490013 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kckt9\" (UniqueName: \"kubernetes.io/projected/4075a738-d882-4257-9997-593dfc82570d-kube-api-access-kckt9\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.490241 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.490086 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.490241 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.490116 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.490241 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.490141 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.490241 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.490193 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.490241 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.490212 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-config\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.540700 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.540682 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-krggs" Oct 30 03:58:20.591026 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.590975 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-config\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591026 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591004 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591026 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591022 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591213 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591044 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591213 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591059 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-web-config\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591213 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591074 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591213 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591094 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591213 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591128 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591213 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591151 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591213 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591173 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4075a738-d882-4257-9997-593dfc82570d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591213 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591200 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4075a738-d882-4257-9997-593dfc82570d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591589 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591233 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4075a738-d882-4257-9997-593dfc82570d-config-out\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591589 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591261 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591589 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591284 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591589 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591304 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591589 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591325 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kckt9\" (UniqueName: \"kubernetes.io/projected/4075a738-d882-4257-9997-593dfc82570d-kube-api-access-kckt9\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591589 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591350 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591589 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591378 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591589 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591393 4793 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 03:58:20.591589 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591437 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.591589 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591478 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.592091 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591670 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4075a738-d882-4257-9997-593dfc82570d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.592091 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591793 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.592091 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.591867 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.592930 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.592877 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.593986 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.593401 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.594090 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.594000 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.594154 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.594100 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.594524 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.594459 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4075a738-d882-4257-9997-593dfc82570d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.595109 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.594900 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.595109 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.594953 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-config\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.595109 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.595061 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-web-config\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.596552 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.596507 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.597411 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.596981 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4075a738-d882-4257-9997-593dfc82570d-config-out\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.597411 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.597370 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.597689 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.597667 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.597775 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.597726 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.598087 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.598040 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.598323 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.598297 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4075a738-d882-4257-9997-593dfc82570d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.598454 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.598436 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4075a738-d882-4257-9997-593dfc82570d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.617582 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.617562 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kckt9\" (UniqueName: \"kubernetes.io/projected/4075a738-d882-4257-9997-593dfc82570d-kube-api-access-kckt9\") pod \"prometheus-k8s-0\" (UID: \"4075a738-d882-4257-9997-593dfc82570d\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.656496 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.656477 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:20.983186 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.983156 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 03:58:20.986330 ip-10-0-138-177 kubenswrapper[4793]: W1030 03:58:20.986303 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4075a738_d882_4257_9997_593dfc82570d.slice/crio-3167beae63d83c302ee8c39e8608ee72e877586c60352f79692ff9ca61b061f0 WatchSource:0}: Error finding container 3167beae63d83c302ee8c39e8608ee72e877586c60352f79692ff9ca61b061f0: Status 404 returned error can't find the container with id 3167beae63d83c302ee8c39e8608ee72e877586c60352f79692ff9ca61b061f0 Oct 30 03:58:20.988966 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:20.988940 4793 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:58:21.953734 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:21.953699 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4075a738-d882-4257-9997-593dfc82570d","Type":"ContainerStarted","Data":"3167beae63d83c302ee8c39e8608ee72e877586c60352f79692ff9ca61b061f0"} Oct 30 03:58:22.632723 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:22.632647 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:22.849762 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:22.849722 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:22.958825 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:22.958794 4793 generic.go:356] "Generic (PLEG): container finished" podID="4075a738-d882-4257-9997-593dfc82570d" containerID="c9cf9511af6a2c5f4ebdc072dbe6a8d912941ed428279f4c7813205d178c19c0" exitCode=0 Oct 30 03:58:22.959200 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:22.958832 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4075a738-d882-4257-9997-593dfc82570d","Type":"ContainerDied","Data":"c9cf9511af6a2c5f4ebdc072dbe6a8d912941ed428279f4c7813205d178c19c0"} Oct 30 03:58:24.481759 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:24.481732 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9gghj" Oct 30 03:58:25.705458 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:25.705437 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-v5zp7" Oct 30 03:58:25.971268 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:25.971179 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4075a738-d882-4257-9997-593dfc82570d","Type":"ContainerStarted","Data":"445c7767865ce7237f2273e39e88f23e2100299e479ee007f77567d6f857693a"} Oct 30 03:58:25.971462 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:25.971287 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4075a738-d882-4257-9997-593dfc82570d","Type":"ContainerStarted","Data":"a44371bfac71b15b34c7d660e110954a34f7c28e74ed953680a170411c64a379"} Oct 30 03:58:27.980815 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:27.980781 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4075a738-d882-4257-9997-593dfc82570d","Type":"ContainerStarted","Data":"94efc912cdca10249f02a2a31606db72ae1f6d65305da4a136359e44341814cb"} Oct 30 03:58:27.981179 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:27.980864 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4075a738-d882-4257-9997-593dfc82570d","Type":"ContainerStarted","Data":"8dce8f8766f26b0dc71294d83d305e4da9ab3fa9025bdccd8bc39a4826c37138"} Oct 30 03:58:27.981179 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:27.980903 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4075a738-d882-4257-9997-593dfc82570d","Type":"ContainerStarted","Data":"26e5b22502f50100c8516fe9032641378933f4887def9e7e755a4ff3efef9639"} Oct 30 03:58:27.981179 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:27.980942 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4075a738-d882-4257-9997-593dfc82570d","Type":"ContainerStarted","Data":"0cf20d9f1f4a1e1d655e18d49bf29ce4bde1eda8d4facfb5a0aa964bd50a6901"} Oct 30 03:58:28.063049 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:28.063001 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.355624146 podStartE2EDuration="8.0629895s" totalImagesPullingTime="5.690305225s" totalInitContainerRuntime="17.060129ms" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:58:28.050345036 +0000 UTC m=+31.336219811" watchObservedRunningTime="2026-10-30 03:58:28.0629895 +0000 UTC m=+31.348864274" Oct 30 03:58:28.884385 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:28.884353 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:29.105541 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:29.105507 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:30.657519 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:30.657483 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:38.785698 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:38.785638 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:38.804092 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:38.804068 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:39.027979 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:39.027954 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 03:58:41.217136 ip-10-0-138-177 kubenswrapper[4793]: E1030 03:58:41.217055 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 03:58:47.955806 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:47.955765 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:48.229828 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:48.229741 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:58:49.914113 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:58:49.914072 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:05.300282 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:59:05.300247 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:05.520371 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:59:05.520341 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:11.215896 ip-10-0-138-177 kubenswrapper[4793]: E1030 03:59:11.215863 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 03:59:41.217034 ip-10-0-138-177 kubenswrapper[4793]: E1030 03:59:41.216992 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 03:59:49.934754 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:59:49.934707 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 03:59:55.324061 ip-10-0-138-177 kubenswrapper[4793]: I1030 03:59:55.324029 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:11.217338 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:00:11.217246 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:00:16.143075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:16.143033 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:16.362772 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:16.362738 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:39.614249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:39.614212 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:39.856097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:39.856065 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:41.216876 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:00:41.216831 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:00:41.289342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:41.289312 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:41.513069 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:41.513001 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:42.637796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:42.637761 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:43.327574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.327541 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk"] Oct 30 04:00:43.330874 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.330858 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:00:43.358908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.358890 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 04:00:43.359009 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.358983 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-mvqzc\"" Oct 30 04:00:43.359009 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.358991 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 04:00:43.359580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.359561 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk"] Oct 30 04:00:43.412793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.412772 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwtz6\" (UniqueName: \"kubernetes.io/projected/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-kube-api-access-dwtz6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk\" (UID: \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:00:43.412902 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.412804 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk\" (UID: \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:00:43.412902 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.412844 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk\" (UID: \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:00:43.513863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.513824 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk\" (UID: \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:00:43.513990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.513885 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwtz6\" (UniqueName: \"kubernetes.io/projected/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-kube-api-access-dwtz6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk\" (UID: \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:00:43.513990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.513923 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk\" (UID: \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:00:43.514277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.514258 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk\" (UID: \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:00:43.514313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.514268 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk\" (UID: \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:00:43.537948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.537920 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwtz6\" (UniqueName: \"kubernetes.io/projected/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-kube-api-access-dwtz6\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk\" (UID: \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:00:43.646124 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.646065 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:00:43.949826 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:43.949806 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk"] Oct 30 04:00:43.952106 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:00:43.952083 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7b8f81_38ab_4920_94f5_df0c7ea6bd07.slice/crio-4cb3add9d02eeed7b28d8376bc15fac867a821691e59f37d9294322b7ba1f94c WatchSource:0}: Error finding container 4cb3add9d02eeed7b28d8376bc15fac867a821691e59f37d9294322b7ba1f94c: Status 404 returned error can't find the container with id 4cb3add9d02eeed7b28d8376bc15fac867a821691e59f37d9294322b7ba1f94c Oct 30 04:00:44.382558 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:44.382471 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" event={"ID":"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07","Type":"ContainerStarted","Data":"4cb3add9d02eeed7b28d8376bc15fac867a821691e59f37d9294322b7ba1f94c"} Oct 30 04:00:45.948062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:45.948024 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:45.993386 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:45.993353 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn"] Oct 30 04:00:45.996584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:45.996563 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:00:46.024794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.024767 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-sl94r\"" Oct 30 04:00:46.025596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.025577 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn"] Oct 30 04:00:46.107596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.107571 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr"] Oct 30 04:00:46.111036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.111014 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:00:46.126620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.126567 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-5gll5\"" Oct 30 04:00:46.131963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.131943 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr"] Oct 30 04:00:46.136528 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.136506 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn\" (UID: \"2443d542-1f1c-481e-9f46-0dd99e699f14\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:00:46.136629 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.136593 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6snv\" (UniqueName: \"kubernetes.io/projected/2443d542-1f1c-481e-9f46-0dd99e699f14-kube-api-access-j6snv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn\" (UID: \"2443d542-1f1c-481e-9f46-0dd99e699f14\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:00:46.136681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.136648 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn\" (UID: \"2443d542-1f1c-481e-9f46-0dd99e699f14\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:00:46.182897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.182863 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:46.212709 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.212650 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz"] Oct 30 04:00:46.215918 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.215898 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:00:46.234810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.233955 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-n6fr2\"" Oct 30 04:00:46.234810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.234758 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz"] Oct 30 04:00:46.237459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.237411 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc47r\" (UniqueName: \"kubernetes.io/projected/c1803776-6697-416a-b637-0c4af5e727ad-kube-api-access-rc47r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr\" (UID: \"c1803776-6697-416a-b637-0c4af5e727ad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:00:46.237549 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.237517 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn\" (UID: \"2443d542-1f1c-481e-9f46-0dd99e699f14\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:00:46.237549 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.237545 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr\" (UID: \"c1803776-6697-416a-b637-0c4af5e727ad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:00:46.237626 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.237567 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr\" (UID: \"c1803776-6697-416a-b637-0c4af5e727ad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:00:46.237626 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.237586 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6snv\" (UniqueName: \"kubernetes.io/projected/2443d542-1f1c-481e-9f46-0dd99e699f14-kube-api-access-j6snv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn\" (UID: \"2443d542-1f1c-481e-9f46-0dd99e699f14\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:00:46.237725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.237626 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn\" (UID: \"2443d542-1f1c-481e-9f46-0dd99e699f14\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:00:46.238085 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.238065 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn\" (UID: \"2443d542-1f1c-481e-9f46-0dd99e699f14\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:00:46.238173 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.238111 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn\" (UID: \"2443d542-1f1c-481e-9f46-0dd99e699f14\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:00:46.261037 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.261013 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6snv\" (UniqueName: \"kubernetes.io/projected/2443d542-1f1c-481e-9f46-0dd99e699f14-kube-api-access-j6snv\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn\" (UID: \"2443d542-1f1c-481e-9f46-0dd99e699f14\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:00:46.295006 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.294985 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr"] Oct 30 04:00:46.298319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.298300 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:00:46.313796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.313774 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:00:46.317522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.317502 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr"] Oct 30 04:00:46.332027 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.332006 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-f2x8l\"" Oct 30 04:00:46.338399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.338365 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc47r\" (UniqueName: \"kubernetes.io/projected/c1803776-6697-416a-b637-0c4af5e727ad-kube-api-access-rc47r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr\" (UID: \"c1803776-6697-416a-b637-0c4af5e727ad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:00:46.338399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.338433 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr\" (UID: \"c1803776-6697-416a-b637-0c4af5e727ad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:00:46.338399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.338462 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr\" (UID: \"c1803776-6697-416a-b637-0c4af5e727ad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:00:46.338910 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.338501 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz\" (UID: \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:00:46.338910 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.338536 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85zfb\" (UniqueName: \"kubernetes.io/projected/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-kube-api-access-85zfb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz\" (UID: \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:00:46.338910 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.338575 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz\" (UID: \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:00:46.339041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.338956 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr\" (UID: \"c1803776-6697-416a-b637-0c4af5e727ad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:00:46.339041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.338981 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr\" (UID: \"c1803776-6697-416a-b637-0c4af5e727ad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:00:46.366401 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.366374 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc47r\" (UniqueName: \"kubernetes.io/projected/c1803776-6697-416a-b637-0c4af5e727ad-kube-api-access-rc47r\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr\" (UID: \"c1803776-6697-416a-b637-0c4af5e727ad\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:00:46.389122 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.389088 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8"] Oct 30 04:00:46.398930 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.397950 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.413867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.413840 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8"] Oct 30 04:00:46.414655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.414595 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vwdzb\"" Oct 30 04:00:46.414655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.414603 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 04:00:46.414655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.414603 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 04:00:46.414861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.414659 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 04:00:46.426760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.426739 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:00:46.439972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.439944 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpn5h\" (UniqueName: \"kubernetes.io/projected/214cc763-e0ff-4de8-9c96-560828f55560-kube-api-access-hpn5h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr\" (UID: \"214cc763-e0ff-4de8-9c96-560828f55560\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:00:46.440086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.440002 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz\" (UID: \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:00:46.440086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.440027 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85zfb\" (UniqueName: \"kubernetes.io/projected/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-kube-api-access-85zfb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz\" (UID: \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:00:46.440086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.440069 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz\" (UID: \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:00:46.440240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.440100 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr\" (UID: \"214cc763-e0ff-4de8-9c96-560828f55560\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:00:46.440240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.440144 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr\" (UID: \"214cc763-e0ff-4de8-9c96-560828f55560\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:00:46.440564 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.440547 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz\" (UID: \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:00:46.441034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.441013 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz\" (UID: \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:00:46.465103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.465046 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85zfb\" (UniqueName: \"kubernetes.io/projected/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-kube-api-access-85zfb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz\" (UID: \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:00:46.533757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.533724 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:00:46.541196 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.541168 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.541323 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.541217 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr\" (UID: \"214cc763-e0ff-4de8-9c96-560828f55560\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:00:46.541628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.541602 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55r6s\" (UniqueName: \"kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-kube-api-access-55r6s\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.541746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.541657 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpn5h\" (UniqueName: \"kubernetes.io/projected/214cc763-e0ff-4de8-9c96-560828f55560-kube-api-access-hpn5h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr\" (UID: \"214cc763-e0ff-4de8-9c96-560828f55560\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:00:46.541810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.541747 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.541810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.541797 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.541938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.541822 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr\" (UID: \"214cc763-e0ff-4de8-9c96-560828f55560\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:00:46.541938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.541830 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734b5f28-3d4f-4674-b509-0fb356ee779b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.542037 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.541939 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.542037 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.542021 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.542237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.542052 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr\" (UID: \"214cc763-e0ff-4de8-9c96-560828f55560\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:00:46.542237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.542086 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.542237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.542112 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.542491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.542472 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr\" (UID: \"214cc763-e0ff-4de8-9c96-560828f55560\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:00:46.554644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.554624 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66"] Oct 30 04:00:46.558473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.558450 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.579784 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.579754 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-ks5cj\"" Oct 30 04:00:46.582631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.582382 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpn5h\" (UniqueName: \"kubernetes.io/projected/214cc763-e0ff-4de8-9c96-560828f55560-kube-api-access-hpn5h\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr\" (UID: \"214cc763-e0ff-4de8-9c96-560828f55560\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:00:46.586130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.586096 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66"] Oct 30 04:00:46.600765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.600671 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:46.615913 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.615893 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:00:46.624526 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.624496 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn"] Oct 30 04:00:46.642741 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.642718 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55r6s\" (UniqueName: \"kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-kube-api-access-55r6s\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.642958 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.642894 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990707cc-7db4-490b-a520-9af03daeeeff-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.643125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.643103 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.643201 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.643142 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990707cc-7db4-490b-a520-9af03daeeeff-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.643201 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.643169 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4kqv\" (UniqueName: \"kubernetes.io/projected/990707cc-7db4-490b-a520-9af03daeeeff-kube-api-access-t4kqv\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.643301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.643205 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.643301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.643282 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.643673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.643649 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734b5f28-3d4f-4674-b509-0fb356ee779b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.643753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.643722 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.643799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.643770 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.643799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.643771 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.644382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.643835 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.644382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.643900 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990707cc-7db4-490b-a520-9af03daeeeff-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.644382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.643986 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.644382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.644020 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.644382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.644125 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.644382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.644157 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.644382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.644192 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.644382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.644219 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.644382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.644225 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.644382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.644259 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.644999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.644486 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734b5f28-3d4f-4674-b509-0fb356ee779b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.644999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.644560 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.646732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.646702 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.646889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.646863 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.666921 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.666895 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55r6s\" (UniqueName: \"kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-kube-api-access-55r6s\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.669447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.668843 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-9vqv8\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.719596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.719539 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:46.733176 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.733139 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr"] Oct 30 04:00:46.736183 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:00:46.736163 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1803776_6697_416a_b637_0c4af5e727ad.slice/crio-04081e579c471681dc07543cd83dbb430c1fec0639b889577d3ce0a5a281ea51 WatchSource:0}: Error finding container 04081e579c471681dc07543cd83dbb430c1fec0639b889577d3ce0a5a281ea51: Status 404 returned error can't find the container with id 04081e579c471681dc07543cd83dbb430c1fec0639b889577d3ce0a5a281ea51 Oct 30 04:00:46.745659 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.745631 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.745753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.745684 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990707cc-7db4-490b-a520-9af03daeeeff-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.745753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.745721 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.745753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.745745 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990707cc-7db4-490b-a520-9af03daeeeff-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.745960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.745772 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4kqv\" (UniqueName: \"kubernetes.io/projected/990707cc-7db4-490b-a520-9af03daeeeff-kube-api-access-t4kqv\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.745960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.745841 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.745960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.745866 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990707cc-7db4-490b-a520-9af03daeeeff-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.745960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.745919 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.745960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.745952 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.746199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.746037 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.746199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.746115 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.746350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.746267 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.746737 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.746704 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990707cc-7db4-490b-a520-9af03daeeeff-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.746827 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.746732 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.748826 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.748801 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990707cc-7db4-490b-a520-9af03daeeeff-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.749190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.749164 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990707cc-7db4-490b-a520-9af03daeeeff-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.769999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.769970 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4kqv\" (UniqueName: \"kubernetes.io/projected/990707cc-7db4-490b-a520-9af03daeeeff-kube-api-access-t4kqv\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.770133 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.770116 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990707cc-7db4-490b-a520-9af03daeeeff-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66\" (UID: \"990707cc-7db4-490b-a520-9af03daeeeff\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.818105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.818028 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:46.843239 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.843212 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz"] Oct 30 04:00:46.846009 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:00:46.845979 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fd8b9a_47f1_4e8d_a318_ab6e923ba472.slice/crio-6bb67dbb1e3970c61a52cc1cbb98641ca090199fe259fbfe81ae696d019ee31d WatchSource:0}: Error finding container 6bb67dbb1e3970c61a52cc1cbb98641ca090199fe259fbfe81ae696d019ee31d: Status 404 returned error can't find the container with id 6bb67dbb1e3970c61a52cc1cbb98641ca090199fe259fbfe81ae696d019ee31d Oct 30 04:00:46.878536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.878513 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:46.918217 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:46.918188 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr"] Oct 30 04:00:46.922540 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:00:46.922514 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214cc763_e0ff_4de8_9c96_560828f55560.slice/crio-34265537fedbf0d8b12456f88c778dcc0eb00fa702afff3ba4236588916d925b WatchSource:0}: Error finding container 34265537fedbf0d8b12456f88c778dcc0eb00fa702afff3ba4236588916d925b: Status 404 returned error can't find the container with id 34265537fedbf0d8b12456f88c778dcc0eb00fa702afff3ba4236588916d925b Oct 30 04:00:47.031471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.031263 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8"] Oct 30 04:00:47.034590 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:00:47.034526 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734b5f28_3d4f_4674_b509_0fb356ee779b.slice/crio-5664b309cf46bbd004efd90323238f6aba7582c091c373bfde9105db2b915361 WatchSource:0}: Error finding container 5664b309cf46bbd004efd90323238f6aba7582c091c373bfde9105db2b915361: Status 404 returned error can't find the container with id 5664b309cf46bbd004efd90323238f6aba7582c091c373bfde9105db2b915361 Oct 30 04:00:47.183621 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.183474 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66"] Oct 30 04:00:47.185787 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:00:47.185754 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990707cc_7db4_490b_a520_9af03daeeeff.slice/crio-ce9f397d8b0cd422bb59d2703e39b78f34b30543a09af2ab457c2cc38cc93285 WatchSource:0}: Error finding container ce9f397d8b0cd422bb59d2703e39b78f34b30543a09af2ab457c2cc38cc93285: Status 404 returned error can't find the container with id ce9f397d8b0cd422bb59d2703e39b78f34b30543a09af2ab457c2cc38cc93285 Oct 30 04:00:47.398518 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.398358 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" event={"ID":"2443d542-1f1c-481e-9f46-0dd99e699f14","Type":"ContainerStarted","Data":"b8563e1ed54fbbce5939a48d2c97ff4d8ac6bcba8fd5bcbef75c948d462ba607"} Oct 30 04:00:47.401193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.401096 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" event={"ID":"990707cc-7db4-490b-a520-9af03daeeeff","Type":"ContainerStarted","Data":"ce9f397d8b0cd422bb59d2703e39b78f34b30543a09af2ab457c2cc38cc93285"} Oct 30 04:00:47.405592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.405534 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" event={"ID":"734b5f28-3d4f-4674-b509-0fb356ee779b","Type":"ContainerStarted","Data":"5664b309cf46bbd004efd90323238f6aba7582c091c373bfde9105db2b915361"} Oct 30 04:00:47.406858 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.406823 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" event={"ID":"214cc763-e0ff-4de8-9c96-560828f55560","Type":"ContainerStarted","Data":"34265537fedbf0d8b12456f88c778dcc0eb00fa702afff3ba4236588916d925b"} Oct 30 04:00:47.408400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.408325 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" event={"ID":"30fd8b9a-47f1-4e8d-a318-ab6e923ba472","Type":"ContainerStarted","Data":"6bb67dbb1e3970c61a52cc1cbb98641ca090199fe259fbfe81ae696d019ee31d"} Oct 30 04:00:47.410041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.410017 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" event={"ID":"c1803776-6697-416a-b637-0c4af5e727ad","Type":"ContainerStarted","Data":"04081e579c471681dc07543cd83dbb430c1fec0639b889577d3ce0a5a281ea51"} Oct 30 04:00:47.556848 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.555485 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg"] Oct 30 04:00:47.559450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.559033 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:00:47.578086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.577442 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-bnbhg\"" Oct 30 04:00:47.578086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.578056 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg"] Oct 30 04:00:47.653635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.653560 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hgbc\" (UniqueName: \"kubernetes.io/projected/48efb046-fbed-4f69-a3f7-ee22dea1e87f-kube-api-access-2hgbc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg\" (UID: \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:00:47.653795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.653663 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg\" (UID: \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:00:47.653795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.653692 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg\" (UID: \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:00:47.755820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.754902 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg\" (UID: \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:00:47.755820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.754998 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hgbc\" (UniqueName: \"kubernetes.io/projected/48efb046-fbed-4f69-a3f7-ee22dea1e87f-kube-api-access-2hgbc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg\" (UID: \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:00:47.755820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.755060 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg\" (UID: \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:00:47.755820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.755481 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg\" (UID: \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:00:47.755820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.755770 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg\" (UID: \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:00:47.801498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.801468 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hgbc\" (UniqueName: \"kubernetes.io/projected/48efb046-fbed-4f69-a3f7-ee22dea1e87f-kube-api-access-2hgbc\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg\" (UID: \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:00:47.881584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:47.881293 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:00:48.131663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.130191 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt"] Oct 30 04:00:48.137391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.136662 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:00:48.154546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.154500 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-rhdgb\"" Oct 30 04:00:48.168400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.168368 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt"] Oct 30 04:00:48.210707 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.210646 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg"] Oct 30 04:00:48.221830 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:00:48.214655 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48efb046_fbed_4f69_a3f7_ee22dea1e87f.slice/crio-0ad797e0849f4dcea75057baa066cd18fe8603c8c31a2387a8861070e02bd164 WatchSource:0}: Error finding container 0ad797e0849f4dcea75057baa066cd18fe8603c8c31a2387a8861070e02bd164: Status 404 returned error can't find the container with id 0ad797e0849f4dcea75057baa066cd18fe8603c8c31a2387a8861070e02bd164 Oct 30 04:00:48.260394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.260337 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m566c\" (UniqueName: \"kubernetes.io/projected/3696d7c9-99f7-489f-baaa-5d6ae8899d67-kube-api-access-m566c\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt\" (UID: \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:00:48.260526 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.260405 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt\" (UID: \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:00:48.260526 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.260446 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt\" (UID: \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:00:48.361517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.361472 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m566c\" (UniqueName: \"kubernetes.io/projected/3696d7c9-99f7-489f-baaa-5d6ae8899d67-kube-api-access-m566c\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt\" (UID: \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:00:48.361693 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.361563 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt\" (UID: \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:00:48.361693 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.361587 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt\" (UID: \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:00:48.362072 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.362047 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt\" (UID: \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:00:48.362696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.362670 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt\" (UID: \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:00:48.390294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.390238 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m566c\" (UniqueName: \"kubernetes.io/projected/3696d7c9-99f7-489f-baaa-5d6ae8899d67-kube-api-access-m566c\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt\" (UID: \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:00:48.417246 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.417212 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" event={"ID":"48efb046-fbed-4f69-a3f7-ee22dea1e87f","Type":"ContainerStarted","Data":"0ad797e0849f4dcea75057baa066cd18fe8603c8c31a2387a8861070e02bd164"} Oct 30 04:00:48.462772 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.462702 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:00:48.852739 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:48.852598 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt"] Oct 30 04:00:48.858378 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:00:48.858346 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3696d7c9_99f7_489f_baaa_5d6ae8899d67.slice/crio-b59f7a6cc2f857bc0571a030906967817f38d5e2b2e59c18d4d7423cf774be89 WatchSource:0}: Error finding container b59f7a6cc2f857bc0571a030906967817f38d5e2b2e59c18d4d7423cf774be89: Status 404 returned error can't find the container with id b59f7a6cc2f857bc0571a030906967817f38d5e2b2e59c18d4d7423cf774be89 Oct 30 04:00:49.429513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:49.429367 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" event={"ID":"3696d7c9-99f7-489f-baaa-5d6ae8899d67","Type":"ContainerStarted","Data":"b59f7a6cc2f857bc0571a030906967817f38d5e2b2e59c18d4d7423cf774be89"} Oct 30 04:00:50.057540 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:50.057450 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:50.285193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:50.284830 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:00:55.191935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:55.191894 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:00:55.192254 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:55.191987 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:00:55.192254 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:55.192034 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:00:55.399516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:55.398841 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:00:55.399516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:55.398935 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:00:55.399516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:55.398984 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:00:55.475127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:55.474411 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" event={"ID":"734b5f28-3d4f-4674-b509-0fb356ee779b","Type":"ContainerStarted","Data":"80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8"} Oct 30 04:00:55.720393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:55.720263 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:56.483904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:56.482641 4793 generic.go:356] "Generic (PLEG): container finished" podID="0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" containerID="a7485d61cf908b9c7682a060c135bec9bc7062348f4c677f1090a50815df71fe" exitCode=0 Oct 30 04:00:56.483904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:56.482707 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" event={"ID":"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07","Type":"ContainerDied","Data":"a7485d61cf908b9c7682a060c135bec9bc7062348f4c677f1090a50815df71fe"} Oct 30 04:00:56.488819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:56.488791 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" event={"ID":"990707cc-7db4-490b-a520-9af03daeeeff","Type":"ContainerStarted","Data":"aac7bd668af565591d7b83966b64e5c305b3bcfb4a54649d177c3d95e4c93bd9"} Oct 30 04:00:56.494947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:56.494887 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" event={"ID":"c1803776-6697-416a-b637-0c4af5e727ad","Type":"ContainerStarted","Data":"3b1ced375fd6c5cea43195ab5299a823e1fcd40e260dc90a379000d5c92a16d9"} Oct 30 04:00:56.538249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:56.537781 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" podStartSLOduration=2.383579051 podStartE2EDuration="10.53776317s" totalImagesPullingTime="8.154184119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:55.536401422 +0000 UTC m=+178.822276199" watchObservedRunningTime="2026-10-30 04:00:56.53776317 +0000 UTC m=+179.823637946" Oct 30 04:00:56.628100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:56.627833 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" podStartSLOduration=2.418291119 podStartE2EDuration="10.627818407s" totalImagesPullingTime="8.209527288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:00:56.627150797 +0000 UTC m=+179.913025608" watchObservedRunningTime="2026-10-30 04:00:56.627818407 +0000 UTC m=+179.913693182" Oct 30 04:00:56.726571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:56.726505 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:56.879400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:56.879323 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:56.885598 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:56.885570 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:57.501509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:57.501148 4793 generic.go:356] "Generic (PLEG): container finished" podID="214cc763-e0ff-4de8-9c96-560828f55560" containerID="f4b726fdee96fbd27d4a49447459e7504e20a9c0528e1494897e0ddafa6796cd" exitCode=0 Oct 30 04:00:57.501509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:57.501452 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" event={"ID":"214cc763-e0ff-4de8-9c96-560828f55560","Type":"ContainerDied","Data":"f4b726fdee96fbd27d4a49447459e7504e20a9c0528e1494897e0ddafa6796cd"} Oct 30 04:00:57.505101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:57.504211 4793 generic.go:356] "Generic (PLEG): container finished" podID="30fd8b9a-47f1-4e8d-a318-ab6e923ba472" containerID="5aa6dad8d87cb5f8da52af4499aa2fe104474d7536954b14cc77501dd6128c32" exitCode=0 Oct 30 04:00:57.505101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:57.504356 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" event={"ID":"30fd8b9a-47f1-4e8d-a318-ab6e923ba472","Type":"ContainerDied","Data":"5aa6dad8d87cb5f8da52af4499aa2fe104474d7536954b14cc77501dd6128c32"} Oct 30 04:00:57.509679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:57.509603 4793 generic.go:356] "Generic (PLEG): container finished" podID="c1803776-6697-416a-b637-0c4af5e727ad" containerID="3b1ced375fd6c5cea43195ab5299a823e1fcd40e260dc90a379000d5c92a16d9" exitCode=0 Oct 30 04:00:57.511151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:57.511128 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" event={"ID":"c1803776-6697-416a-b637-0c4af5e727ad","Type":"ContainerDied","Data":"3b1ced375fd6c5cea43195ab5299a823e1fcd40e260dc90a379000d5c92a16d9"} Oct 30 04:00:57.511252 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:57.511176 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:57.512140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:57.512114 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:57.512364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:57.512347 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66" Oct 30 04:00:57.513258 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:57.513235 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:00:58.517565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:58.517543 4793 generic.go:356] "Generic (PLEG): container finished" podID="48efb046-fbed-4f69-a3f7-ee22dea1e87f" containerID="9cf5a725799642dba496ee0a7797a8b7ae38eff74703ae0f1f87763922ef1532" exitCode=0 Oct 30 04:00:58.517924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:58.517629 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" event={"ID":"48efb046-fbed-4f69-a3f7-ee22dea1e87f","Type":"ContainerDied","Data":"9cf5a725799642dba496ee0a7797a8b7ae38eff74703ae0f1f87763922ef1532"} Oct 30 04:00:58.519900 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:58.519871 4793 generic.go:356] "Generic (PLEG): container finished" podID="3696d7c9-99f7-489f-baaa-5d6ae8899d67" containerID="ffde2327388580bb2da787568078f04174d9c689c04ca86ab628f4a2fc792f38" exitCode=0 Oct 30 04:00:58.519993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:58.519963 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" event={"ID":"3696d7c9-99f7-489f-baaa-5d6ae8899d67","Type":"ContainerDied","Data":"ffde2327388580bb2da787568078f04174d9c689c04ca86ab628f4a2fc792f38"} Oct 30 04:00:58.523052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:58.523031 4793 generic.go:356] "Generic (PLEG): container finished" podID="214cc763-e0ff-4de8-9c96-560828f55560" containerID="c43d713c292fae7d5f9390ca89cddc1debbe827c34fa79df80ce2a55c294175d" exitCode=0 Oct 30 04:00:58.523150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:58.523095 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" event={"ID":"214cc763-e0ff-4de8-9c96-560828f55560","Type":"ContainerDied","Data":"c43d713c292fae7d5f9390ca89cddc1debbe827c34fa79df80ce2a55c294175d"} Oct 30 04:00:58.529876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:58.529847 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" event={"ID":"30fd8b9a-47f1-4e8d-a318-ab6e923ba472","Type":"ContainerStarted","Data":"cd6d1601f7e449a88780a0eea43ec3ffb606cbfa0a0548ef40b9ba1cff2811de"} Oct 30 04:00:59.535322 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:59.535276 4793 generic.go:356] "Generic (PLEG): container finished" podID="30fd8b9a-47f1-4e8d-a318-ab6e923ba472" containerID="cd6d1601f7e449a88780a0eea43ec3ffb606cbfa0a0548ef40b9ba1cff2811de" exitCode=0 Oct 30 04:00:59.535755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:59.535375 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" event={"ID":"30fd8b9a-47f1-4e8d-a318-ab6e923ba472","Type":"ContainerDied","Data":"cd6d1601f7e449a88780a0eea43ec3ffb606cbfa0a0548ef40b9ba1cff2811de"} Oct 30 04:00:59.538661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:59.538635 4793 generic.go:356] "Generic (PLEG): container finished" podID="c1803776-6697-416a-b637-0c4af5e727ad" containerID="f755ad88ba591c8bd0046c6f5bebbc3ce21d7a6e612683b26b26655a75e600f6" exitCode=0 Oct 30 04:00:59.538766 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:59.538725 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" event={"ID":"c1803776-6697-416a-b637-0c4af5e727ad","Type":"ContainerDied","Data":"f755ad88ba591c8bd0046c6f5bebbc3ce21d7a6e612683b26b26655a75e600f6"} Oct 30 04:00:59.541007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:59.540981 4793 generic.go:356] "Generic (PLEG): container finished" podID="2443d542-1f1c-481e-9f46-0dd99e699f14" containerID="a5765d1feddb0d6c1312fa6c1d9203f75352efb88bf2225a81dd544db42e2c62" exitCode=0 Oct 30 04:00:59.541119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:59.541104 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" event={"ID":"2443d542-1f1c-481e-9f46-0dd99e699f14","Type":"ContainerDied","Data":"a5765d1feddb0d6c1312fa6c1d9203f75352efb88bf2225a81dd544db42e2c62"} Oct 30 04:00:59.544026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:59.544002 4793 generic.go:356] "Generic (PLEG): container finished" podID="0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" containerID="d19d72efb4018ad9d99d591d7adcc6e4da70fc86add46f2da4f9f765b6482fd0" exitCode=0 Oct 30 04:00:59.544132 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:59.544082 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" event={"ID":"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07","Type":"ContainerDied","Data":"d19d72efb4018ad9d99d591d7adcc6e4da70fc86add46f2da4f9f765b6482fd0"} Oct 30 04:00:59.546928 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:59.546905 4793 generic.go:356] "Generic (PLEG): container finished" podID="48efb046-fbed-4f69-a3f7-ee22dea1e87f" containerID="869389995fd48a83b46617280406f0b5b0335f38ce913c1069e673897eaa6e1d" exitCode=0 Oct 30 04:00:59.547021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:00:59.546943 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" event={"ID":"48efb046-fbed-4f69-a3f7-ee22dea1e87f","Type":"ContainerDied","Data":"869389995fd48a83b46617280406f0b5b0335f38ce913c1069e673897eaa6e1d"} Oct 30 04:01:01.564012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:01.563973 4793 generic.go:356] "Generic (PLEG): container finished" podID="2443d542-1f1c-481e-9f46-0dd99e699f14" containerID="1a9648ee6fc102e63f5110d22efadbcde9ba1267fc0f864273996c9890f4b5b1" exitCode=0 Oct 30 04:01:01.564485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:01.564066 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" event={"ID":"2443d542-1f1c-481e-9f46-0dd99e699f14","Type":"ContainerDied","Data":"1a9648ee6fc102e63f5110d22efadbcde9ba1267fc0f864273996c9890f4b5b1"} Oct 30 04:01:04.606627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:04.605650 4793 generic.go:356] "Generic (PLEG): container finished" podID="3696d7c9-99f7-489f-baaa-5d6ae8899d67" containerID="b8ec785f60277c58c8527088196a1fc66fd7833b24a5fb11e91b9ec3d7296bde" exitCode=0 Oct 30 04:01:04.606627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:04.605696 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" event={"ID":"3696d7c9-99f7-489f-baaa-5d6ae8899d67","Type":"ContainerDied","Data":"b8ec785f60277c58c8527088196a1fc66fd7833b24a5fb11e91b9ec3d7296bde"} Oct 30 04:01:11.247138 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:01:11.247019 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:01:12.657798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:12.657226 4793 generic.go:356] "Generic (PLEG): container finished" podID="214cc763-e0ff-4de8-9c96-560828f55560" containerID="50fbe90c6bb8dc7e9a0fa0e3156cb93a8e5cb8644d6d89cb5576f64119de1417" exitCode=0 Oct 30 04:01:12.657798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:12.657357 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" event={"ID":"214cc763-e0ff-4de8-9c96-560828f55560","Type":"ContainerDied","Data":"50fbe90c6bb8dc7e9a0fa0e3156cb93a8e5cb8644d6d89cb5576f64119de1417"} Oct 30 04:01:13.844167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:13.844144 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:01:13.915844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:13.915822 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/214cc763-e0ff-4de8-9c96-560828f55560-kube-api-access-hpn5h\" (UniqueName: \"kubernetes.io/projected/214cc763-e0ff-4de8-9c96-560828f55560-kube-api-access-hpn5h\") pod \"214cc763-e0ff-4de8-9c96-560828f55560\" (UID: \"214cc763-e0ff-4de8-9c96-560828f55560\") " Oct 30 04:01:13.915992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:13.915970 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-bundle\" (UniqueName: \"kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-bundle\") pod \"214cc763-e0ff-4de8-9c96-560828f55560\" (UID: \"214cc763-e0ff-4de8-9c96-560828f55560\") " Oct 30 04:01:13.916060 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:13.916009 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-util\" (UniqueName: \"kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-util\") pod \"214cc763-e0ff-4de8-9c96-560828f55560\" (UID: \"214cc763-e0ff-4de8-9c96-560828f55560\") " Oct 30 04:01:13.916871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:13.916816 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-bundle" pod "214cc763-e0ff-4de8-9c96-560828f55560" (UID: "214cc763-e0ff-4de8-9c96-560828f55560"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:13.919781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:13.919757 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/214cc763-e0ff-4de8-9c96-560828f55560-kube-api-access-hpn5h" pod "214cc763-e0ff-4de8-9c96-560828f55560" (UID: "214cc763-e0ff-4de8-9c96-560828f55560"). InnerVolumeSpecName "kube-api-access-hpn5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:01:13.923753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:13.923731 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-util" pod "214cc763-e0ff-4de8-9c96-560828f55560" (UID: "214cc763-e0ff-4de8-9c96-560828f55560"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:14.017208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.017184 4793 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-bundle\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:14.017208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.017207 4793 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/214cc763-e0ff-4de8-9c96-560828f55560-util\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:14.017449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.017217 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpn5h\" (UniqueName: \"kubernetes.io/projected/214cc763-e0ff-4de8-9c96-560828f55560-kube-api-access-hpn5h\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:14.677537 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.677396 4793 generic.go:356] "Generic (PLEG): container finished" podID="0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" containerID="3c610537488485b1bdb445753e5866d5e89624af7a3ed8f46cac1e5fe7fe7d80" exitCode=0 Oct 30 04:01:14.677537 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.677459 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" event={"ID":"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07","Type":"ContainerDied","Data":"3c610537488485b1bdb445753e5866d5e89624af7a3ed8f46cac1e5fe7fe7d80"} Oct 30 04:01:14.679934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.679907 4793 generic.go:356] "Generic (PLEG): container finished" podID="48efb046-fbed-4f69-a3f7-ee22dea1e87f" containerID="827b3a348353630cf8899417d0431777de6154ff708f6c90f4e2f5167a1c1e5d" exitCode=0 Oct 30 04:01:14.680063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.679976 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" event={"ID":"48efb046-fbed-4f69-a3f7-ee22dea1e87f","Type":"ContainerDied","Data":"827b3a348353630cf8899417d0431777de6154ff708f6c90f4e2f5167a1c1e5d"} Oct 30 04:01:14.682381 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.682353 4793 generic.go:356] "Generic (PLEG): container finished" podID="3696d7c9-99f7-489f-baaa-5d6ae8899d67" containerID="2ecd8752d3bd2dcc5cc7a05b6bac96f5d2b3ea3e8e455abd1ba9901216bcf5ad" exitCode=0 Oct 30 04:01:14.682538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.682451 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" event={"ID":"3696d7c9-99f7-489f-baaa-5d6ae8899d67","Type":"ContainerDied","Data":"2ecd8752d3bd2dcc5cc7a05b6bac96f5d2b3ea3e8e455abd1ba9901216bcf5ad"} Oct 30 04:01:14.684494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.684459 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" event={"ID":"214cc763-e0ff-4de8-9c96-560828f55560","Type":"ContainerDied","Data":"34265537fedbf0d8b12456f88c778dcc0eb00fa702afff3ba4236588916d925b"} Oct 30 04:01:14.684631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.684515 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34265537fedbf0d8b12456f88c778dcc0eb00fa702afff3ba4236588916d925b" Oct 30 04:01:14.684631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.684473 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c255qr" Oct 30 04:01:14.686945 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.686867 4793 generic.go:356] "Generic (PLEG): container finished" podID="30fd8b9a-47f1-4e8d-a318-ab6e923ba472" containerID="36d01fe19e49ea9c3a38d388d5e8c289192368738b714c6c73558ed6e27198c7" exitCode=0 Oct 30 04:01:14.687058 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.686937 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" event={"ID":"30fd8b9a-47f1-4e8d-a318-ab6e923ba472","Type":"ContainerDied","Data":"36d01fe19e49ea9c3a38d388d5e8c289192368738b714c6c73558ed6e27198c7"} Oct 30 04:01:14.689178 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.689158 4793 generic.go:356] "Generic (PLEG): container finished" podID="c1803776-6697-416a-b637-0c4af5e727ad" containerID="bf49f33fcc74b31517395960876b631e0166da497cea7fa53dd9db4f9b5aff94" exitCode=0 Oct 30 04:01:14.689297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:14.689200 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" event={"ID":"c1803776-6697-416a-b637-0c4af5e727ad","Type":"ContainerDied","Data":"bf49f33fcc74b31517395960876b631e0166da497cea7fa53dd9db4f9b5aff94"} Oct 30 04:01:15.694653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.694626 4793 generic.go:356] "Generic (PLEG): container finished" podID="2443d542-1f1c-481e-9f46-0dd99e699f14" containerID="88f5d02c0058086a0685550c5920f6cfc62860cfcb0c0d5df75eb21f6a74fb14" exitCode=0 Oct 30 04:01:15.694996 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.694720 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" event={"ID":"2443d542-1f1c-481e-9f46-0dd99e699f14","Type":"ContainerDied","Data":"88f5d02c0058086a0685550c5920f6cfc62860cfcb0c0d5df75eb21f6a74fb14"} Oct 30 04:01:15.850151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.850128 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:01:15.910984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.910964 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:01:15.914087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.914067 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:01:15.917208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.917187 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:01:15.920163 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.920145 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:01:15.935041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.935007 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-util\" (UniqueName: \"kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-util\") pod \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\" (UID: \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\") " Oct 30 04:01:15.935142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.935123 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-bundle\" (UniqueName: \"kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-bundle\") pod \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\" (UID: \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\") " Oct 30 04:01:15.935204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.935178 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-kube-api-access-85zfb\" (UniqueName: \"kubernetes.io/projected/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-kube-api-access-85zfb\") pod \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\" (UID: \"30fd8b9a-47f1-4e8d-a318-ab6e923ba472\") " Oct 30 04:01:15.935556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.935522 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-bundle" pod "30fd8b9a-47f1-4e8d-a318-ab6e923ba472" (UID: "30fd8b9a-47f1-4e8d-a318-ab6e923ba472"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:15.937139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.937116 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-kube-api-access-85zfb" pod "30fd8b9a-47f1-4e8d-a318-ab6e923ba472" (UID: "30fd8b9a-47f1-4e8d-a318-ab6e923ba472"). InnerVolumeSpecName "kube-api-access-85zfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:01:15.939876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:15.939857 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-util" pod "30fd8b9a-47f1-4e8d-a318-ab6e923ba472" (UID: "30fd8b9a-47f1-4e8d-a318-ab6e923ba472"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:16.036348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036274 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-util\" (UniqueName: \"kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-util\") pod \"c1803776-6697-416a-b637-0c4af5e727ad\" (UID: \"c1803776-6697-416a-b637-0c4af5e727ad\") " Oct 30 04:01:16.036348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036310 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-bundle\") pod \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\" (UID: \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\") " Oct 30 04:01:16.036348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036336 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-util\" (UniqueName: \"kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-util\") pod \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\" (UID: \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\") " Oct 30 04:01:16.036627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036367 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-util\" (UniqueName: \"kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-util\") pod \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\" (UID: \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\") " Oct 30 04:01:16.036627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036394 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-bundle\") pod \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\" (UID: \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\") " Oct 30 04:01:16.036627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036450 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-bundle\") pod \"c1803776-6697-416a-b637-0c4af5e727ad\" (UID: \"c1803776-6697-416a-b637-0c4af5e727ad\") " Oct 30 04:01:16.036627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036485 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1803776-6697-416a-b637-0c4af5e727ad-kube-api-access-rc47r\" (UniqueName: \"kubernetes.io/projected/c1803776-6697-416a-b637-0c4af5e727ad-kube-api-access-rc47r\") pod \"c1803776-6697-416a-b637-0c4af5e727ad\" (UID: \"c1803776-6697-416a-b637-0c4af5e727ad\") " Oct 30 04:01:16.036627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036524 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48efb046-fbed-4f69-a3f7-ee22dea1e87f-kube-api-access-2hgbc\" (UniqueName: \"kubernetes.io/projected/48efb046-fbed-4f69-a3f7-ee22dea1e87f-kube-api-access-2hgbc\") pod \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\" (UID: \"48efb046-fbed-4f69-a3f7-ee22dea1e87f\") " Oct 30 04:01:16.036627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036561 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-kube-api-access-dwtz6\" (UniqueName: \"kubernetes.io/projected/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-kube-api-access-dwtz6\") pod \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\" (UID: \"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07\") " Oct 30 04:01:16.036627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036588 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3696d7c9-99f7-489f-baaa-5d6ae8899d67-kube-api-access-m566c\" (UniqueName: \"kubernetes.io/projected/3696d7c9-99f7-489f-baaa-5d6ae8899d67-kube-api-access-m566c\") pod \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\" (UID: \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\") " Oct 30 04:01:16.036627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036619 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-util\" (UniqueName: \"kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-util\") pod \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\" (UID: \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\") " Oct 30 04:01:16.037049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036681 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-bundle\") pod \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\" (UID: \"3696d7c9-99f7-489f-baaa-5d6ae8899d67\") " Oct 30 04:01:16.037049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.036864 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-bundle" pod "0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" (UID: "0f7b8f81-38ab-4920-94f5-df0c7ea6bd07"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:16.037188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.037095 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-bundle" pod "48efb046-fbed-4f69-a3f7-ee22dea1e87f" (UID: "48efb046-fbed-4f69-a3f7-ee22dea1e87f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:16.037327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.037297 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-bundle" pod "c1803776-6697-416a-b637-0c4af5e727ad" (UID: "c1803776-6697-416a-b637-0c4af5e727ad"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:16.041472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.039568 4793 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-bundle\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.041472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.039594 4793 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-bundle\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.041472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.039606 4793 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-bundle\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.041472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.039619 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85zfb\" (UniqueName: \"kubernetes.io/projected/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-kube-api-access-85zfb\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.041472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.039635 4793 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-bundle\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.041472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.039647 4793 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/30fd8b9a-47f1-4e8d-a318-ab6e923ba472-util\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.041472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.040767 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-bundle" pod "3696d7c9-99f7-489f-baaa-5d6ae8899d67" (UID: "3696d7c9-99f7-489f-baaa-5d6ae8899d67"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:16.041879 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.041738 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1803776-6697-416a-b637-0c4af5e727ad-kube-api-access-rc47r" pod "c1803776-6697-416a-b637-0c4af5e727ad" (UID: "c1803776-6697-416a-b637-0c4af5e727ad"). InnerVolumeSpecName "kube-api-access-rc47r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:01:16.043454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.042564 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3696d7c9-99f7-489f-baaa-5d6ae8899d67-kube-api-access-m566c" pod "3696d7c9-99f7-489f-baaa-5d6ae8899d67" (UID: "3696d7c9-99f7-489f-baaa-5d6ae8899d67"). InnerVolumeSpecName "kube-api-access-m566c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:01:16.043645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.043615 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48efb046-fbed-4f69-a3f7-ee22dea1e87f-kube-api-access-2hgbc" pod "48efb046-fbed-4f69-a3f7-ee22dea1e87f" (UID: "48efb046-fbed-4f69-a3f7-ee22dea1e87f"). InnerVolumeSpecName "kube-api-access-2hgbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:01:16.044079 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.044041 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-util" pod "3696d7c9-99f7-489f-baaa-5d6ae8899d67" (UID: "3696d7c9-99f7-489f-baaa-5d6ae8899d67"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:16.045044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.045018 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-util" pod "c1803776-6697-416a-b637-0c4af5e727ad" (UID: "c1803776-6697-416a-b637-0c4af5e727ad"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:16.045131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.045059 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-kube-api-access-dwtz6" pod "0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" (UID: "0f7b8f81-38ab-4920-94f5-df0c7ea6bd07"). InnerVolumeSpecName "kube-api-access-dwtz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:01:16.045197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.045176 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-util" pod "0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" (UID: "0f7b8f81-38ab-4920-94f5-df0c7ea6bd07"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:16.045517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.045493 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-util" pod "48efb046-fbed-4f69-a3f7-ee22dea1e87f" (UID: "48efb046-fbed-4f69-a3f7-ee22dea1e87f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:16.139964 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.139922 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hgbc\" (UniqueName: \"kubernetes.io/projected/48efb046-fbed-4f69-a3f7-ee22dea1e87f-kube-api-access-2hgbc\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.139964 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.139962 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwtz6\" (UniqueName: \"kubernetes.io/projected/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-kube-api-access-dwtz6\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.139964 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.139972 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m566c\" (UniqueName: \"kubernetes.io/projected/3696d7c9-99f7-489f-baaa-5d6ae8899d67-kube-api-access-m566c\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.140127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.139982 4793 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-util\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.140127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.139992 4793 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3696d7c9-99f7-489f-baaa-5d6ae8899d67-bundle\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.140127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.139999 4793 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c1803776-6697-416a-b637-0c4af5e727ad-util\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.140127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.140006 4793 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/48efb046-fbed-4f69-a3f7-ee22dea1e87f-util\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.140127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.140014 4793 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0f7b8f81-38ab-4920-94f5-df0c7ea6bd07-util\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.140127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.140021 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc47r\" (UniqueName: \"kubernetes.io/projected/c1803776-6697-416a-b637-0c4af5e727ad-kube-api-access-rc47r\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:16.700131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.700089 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" event={"ID":"30fd8b9a-47f1-4e8d-a318-ab6e923ba472","Type":"ContainerDied","Data":"6bb67dbb1e3970c61a52cc1cbb98641ca090199fe259fbfe81ae696d019ee31d"} Oct 30 04:01:16.700608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.700149 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bb67dbb1e3970c61a52cc1cbb98641ca090199fe259fbfe81ae696d019ee31d" Oct 30 04:01:16.700608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.700105 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6b58fz" Oct 30 04:01:16.702253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.702226 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" Oct 30 04:01:16.702407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.702257 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7nr7rr" event={"ID":"c1803776-6697-416a-b637-0c4af5e727ad","Type":"ContainerDied","Data":"04081e579c471681dc07543cd83dbb430c1fec0639b889577d3ce0a5a281ea51"} Oct 30 04:01:16.702407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.702296 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04081e579c471681dc07543cd83dbb430c1fec0639b889577d3ce0a5a281ea51" Oct 30 04:01:16.704385 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.704358 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" event={"ID":"0f7b8f81-38ab-4920-94f5-df0c7ea6bd07","Type":"ContainerDied","Data":"4cb3add9d02eeed7b28d8376bc15fac867a821691e59f37d9294322b7ba1f94c"} Oct 30 04:01:16.704531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.704397 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cb3add9d02eeed7b28d8376bc15fac867a821691e59f37d9294322b7ba1f94c" Oct 30 04:01:16.704531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.704366 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644pk2bk" Oct 30 04:01:16.706472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.706444 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" event={"ID":"48efb046-fbed-4f69-a3f7-ee22dea1e87f","Type":"ContainerDied","Data":"0ad797e0849f4dcea75057baa066cd18fe8603c8c31a2387a8861070e02bd164"} Oct 30 04:01:16.706619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.706495 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad797e0849f4dcea75057baa066cd18fe8603c8c31a2387a8861070e02bd164" Oct 30 04:01:16.706619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.706449 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be716wvsg" Oct 30 04:01:16.715599 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.715408 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" event={"ID":"3696d7c9-99f7-489f-baaa-5d6ae8899d67","Type":"ContainerDied","Data":"b59f7a6cc2f857bc0571a030906967817f38d5e2b2e59c18d4d7423cf774be89"} Oct 30 04:01:16.715599 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.715566 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b59f7a6cc2f857bc0571a030906967817f38d5e2b2e59c18d4d7423cf774be89" Oct 30 04:01:16.716154 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.716044 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c6t6wt" Oct 30 04:01:16.832067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.832041 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:01:16.945566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.945534 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-util\" (UniqueName: \"kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-util\") pod \"2443d542-1f1c-481e-9f46-0dd99e699f14\" (UID: \"2443d542-1f1c-481e-9f46-0dd99e699f14\") " Oct 30 04:01:16.945722 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.945579 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-bundle\") pod \"2443d542-1f1c-481e-9f46-0dd99e699f14\" (UID: \"2443d542-1f1c-481e-9f46-0dd99e699f14\") " Oct 30 04:01:16.945722 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.945647 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2443d542-1f1c-481e-9f46-0dd99e699f14-kube-api-access-j6snv\" (UniqueName: \"kubernetes.io/projected/2443d542-1f1c-481e-9f46-0dd99e699f14-kube-api-access-j6snv\") pod \"2443d542-1f1c-481e-9f46-0dd99e699f14\" (UID: \"2443d542-1f1c-481e-9f46-0dd99e699f14\") " Oct 30 04:01:16.946156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.946130 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-bundle" pod "2443d542-1f1c-481e-9f46-0dd99e699f14" (UID: "2443d542-1f1c-481e-9f46-0dd99e699f14"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:16.947707 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.947685 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2443d542-1f1c-481e-9f46-0dd99e699f14-kube-api-access-j6snv" pod "2443d542-1f1c-481e-9f46-0dd99e699f14" (UID: "2443d542-1f1c-481e-9f46-0dd99e699f14"). InnerVolumeSpecName "kube-api-access-j6snv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:01:16.953354 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:16.953329 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-util" pod "2443d542-1f1c-481e-9f46-0dd99e699f14" (UID: "2443d542-1f1c-481e-9f46-0dd99e699f14"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:01:17.046308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:17.046286 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6snv\" (UniqueName: \"kubernetes.io/projected/2443d542-1f1c-481e-9f46-0dd99e699f14-kube-api-access-j6snv\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:17.046308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:17.046308 4793 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-util\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:17.046465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:17.046320 4793 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2443d542-1f1c-481e-9f46-0dd99e699f14-bundle\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:01:17.721025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:17.720996 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" Oct 30 04:01:17.721025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:17.721004 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847arq5pn" event={"ID":"2443d542-1f1c-481e-9f46-0dd99e699f14","Type":"ContainerDied","Data":"b8563e1ed54fbbce5939a48d2c97ff4d8ac6bcba8fd5bcbef75c948d462ba607"} Oct 30 04:01:17.721467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:17.721046 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8563e1ed54fbbce5939a48d2c97ff4d8ac6bcba8fd5bcbef75c948d462ba607" Oct 30 04:01:19.453660 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:19.453621 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:20.331857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:20.331818 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:20.559556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:20.559522 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:21.853338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853307 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5"] Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853555 4793 cpu_manager.go:414] "containerMap: removing container" podUID="214cc763-e0ff-4de8-9c96-560828f55560" containerName="util" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853566 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="214cc763-e0ff-4de8-9c96-560828f55560" containerName="util" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853577 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3696d7c9-99f7-489f-baaa-5d6ae8899d67" containerName="pull" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853583 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3696d7c9-99f7-489f-baaa-5d6ae8899d67" containerName="pull" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853589 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2443d542-1f1c-481e-9f46-0dd99e699f14" containerName="extract" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853594 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2443d542-1f1c-481e-9f46-0dd99e699f14" containerName="extract" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853603 4793 cpu_manager.go:414] "containerMap: removing container" podUID="30fd8b9a-47f1-4e8d-a318-ab6e923ba472" containerName="extract" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853608 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30fd8b9a-47f1-4e8d-a318-ab6e923ba472" containerName="extract" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853616 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" containerName="extract" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853621 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" containerName="extract" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853628 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" containerName="util" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853633 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" containerName="util" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853638 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c1803776-6697-416a-b637-0c4af5e727ad" containerName="util" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853642 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1803776-6697-416a-b637-0c4af5e727ad" containerName="util" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853648 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2443d542-1f1c-481e-9f46-0dd99e699f14" containerName="util" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853652 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2443d542-1f1c-481e-9f46-0dd99e699f14" containerName="util" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853659 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" containerName="pull" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853663 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" containerName="pull" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853670 4793 cpu_manager.go:414] "containerMap: removing container" podUID="214cc763-e0ff-4de8-9c96-560828f55560" containerName="pull" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853675 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="214cc763-e0ff-4de8-9c96-560828f55560" containerName="pull" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853680 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2443d542-1f1c-481e-9f46-0dd99e699f14" containerName="pull" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853684 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2443d542-1f1c-481e-9f46-0dd99e699f14" containerName="pull" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853689 4793 cpu_manager.go:414] "containerMap: removing container" podUID="48efb046-fbed-4f69-a3f7-ee22dea1e87f" containerName="extract" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853693 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48efb046-fbed-4f69-a3f7-ee22dea1e87f" containerName="extract" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853703 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3696d7c9-99f7-489f-baaa-5d6ae8899d67" containerName="util" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853707 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3696d7c9-99f7-489f-baaa-5d6ae8899d67" containerName="util" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853713 4793 cpu_manager.go:414] "containerMap: removing container" podUID="30fd8b9a-47f1-4e8d-a318-ab6e923ba472" containerName="pull" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853717 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30fd8b9a-47f1-4e8d-a318-ab6e923ba472" containerName="pull" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853723 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3696d7c9-99f7-489f-baaa-5d6ae8899d67" containerName="extract" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853727 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3696d7c9-99f7-489f-baaa-5d6ae8899d67" containerName="extract" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853734 4793 cpu_manager.go:414] "containerMap: removing container" podUID="30fd8b9a-47f1-4e8d-a318-ab6e923ba472" containerName="util" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853739 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30fd8b9a-47f1-4e8d-a318-ab6e923ba472" containerName="util" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853744 4793 cpu_manager.go:414] "containerMap: removing container" podUID="48efb046-fbed-4f69-a3f7-ee22dea1e87f" containerName="pull" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853749 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48efb046-fbed-4f69-a3f7-ee22dea1e87f" containerName="pull" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853755 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c1803776-6697-416a-b637-0c4af5e727ad" containerName="extract" Oct 30 04:01:21.853740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853760 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1803776-6697-416a-b637-0c4af5e727ad" containerName="extract" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853767 4793 cpu_manager.go:414] "containerMap: removing container" podUID="48efb046-fbed-4f69-a3f7-ee22dea1e87f" containerName="util" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853771 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48efb046-fbed-4f69-a3f7-ee22dea1e87f" containerName="util" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853776 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c1803776-6697-416a-b637-0c4af5e727ad" containerName="pull" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853780 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1803776-6697-416a-b637-0c4af5e727ad" containerName="pull" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853785 4793 cpu_manager.go:414] "containerMap: removing container" podUID="214cc763-e0ff-4de8-9c96-560828f55560" containerName="extract" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853789 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="214cc763-e0ff-4de8-9c96-560828f55560" containerName="extract" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853826 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3696d7c9-99f7-489f-baaa-5d6ae8899d67" containerName="extract" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853835 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1803776-6697-416a-b637-0c4af5e727ad" containerName="extract" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853840 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="214cc763-e0ff-4de8-9c96-560828f55560" containerName="extract" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853845 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2443d542-1f1c-481e-9f46-0dd99e699f14" containerName="extract" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853850 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="30fd8b9a-47f1-4e8d-a318-ab6e923ba472" containerName="extract" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853856 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f7b8f81-38ab-4920-94f5-df0c7ea6bd07" containerName="extract" Oct 30 04:01:21.854904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.853862 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="48efb046-fbed-4f69-a3f7-ee22dea1e87f" containerName="extract" Oct 30 04:01:21.876339 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.876315 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5" Oct 30 04:01:21.884379 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.884355 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5"] Oct 30 04:01:21.895509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.895483 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 04:01:21.895509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.895494 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:01:21.895643 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.895510 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fdkrw\"" Oct 30 04:01:21.980212 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.980188 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r669\" (UniqueName: \"kubernetes.io/projected/e6b32021-61b5-49f5-9af6-6234d5aef6eb-kube-api-access-2r669\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5\" (UID: \"e6b32021-61b5-49f5-9af6-6234d5aef6eb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5" Oct 30 04:01:21.980321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:21.980233 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6b32021-61b5-49f5-9af6-6234d5aef6eb-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5\" (UID: \"e6b32021-61b5-49f5-9af6-6234d5aef6eb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5" Oct 30 04:01:22.081538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:22.081515 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r669\" (UniqueName: \"kubernetes.io/projected/e6b32021-61b5-49f5-9af6-6234d5aef6eb-kube-api-access-2r669\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5\" (UID: \"e6b32021-61b5-49f5-9af6-6234d5aef6eb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5" Oct 30 04:01:22.081611 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:22.081550 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6b32021-61b5-49f5-9af6-6234d5aef6eb-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5\" (UID: \"e6b32021-61b5-49f5-9af6-6234d5aef6eb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5" Oct 30 04:01:22.081979 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:22.081959 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6b32021-61b5-49f5-9af6-6234d5aef6eb-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5\" (UID: \"e6b32021-61b5-49f5-9af6-6234d5aef6eb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5" Oct 30 04:01:22.105227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:22.105176 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r669\" (UniqueName: \"kubernetes.io/projected/e6b32021-61b5-49f5-9af6-6234d5aef6eb-kube-api-access-2r669\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5\" (UID: \"e6b32021-61b5-49f5-9af6-6234d5aef6eb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5" Oct 30 04:01:22.198780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:22.198759 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5" Oct 30 04:01:22.455977 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:22.455939 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:22.539984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:22.539964 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5"] Oct 30 04:01:22.541768 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:01:22.541738 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b32021_61b5_49f5_9af6_6234d5aef6eb.slice/crio-7642305446e26a74faeb34d374beb4b6cf45529643bbbd6be2fea020e468a6c6 WatchSource:0}: Error finding container 7642305446e26a74faeb34d374beb4b6cf45529643bbbd6be2fea020e468a6c6: Status 404 returned error can't find the container with id 7642305446e26a74faeb34d374beb4b6cf45529643bbbd6be2fea020e468a6c6 Oct 30 04:01:22.740572 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:22.740508 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5" event={"ID":"e6b32021-61b5-49f5-9af6-6234d5aef6eb","Type":"ContainerStarted","Data":"7642305446e26a74faeb34d374beb4b6cf45529643bbbd6be2fea020e468a6c6"} Oct 30 04:01:23.871050 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:23.871016 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86"] Oct 30 04:01:23.874102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:23.874071 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:23.904651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:23.904625 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 04:01:23.904651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:23.904642 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 04:01:23.904864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:23.904653 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 04:01:23.904864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:23.904630 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 04:01:23.904864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:23.904718 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-8srh5\"" Oct 30 04:01:23.904864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:23.904772 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 04:01:23.911011 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:23.910988 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86"] Oct 30 04:01:24.006165 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.006129 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a784a221-4d26-4f11-b573-cf1046ea769e-metrics-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.006353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.006172 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vcjd\" (UniqueName: \"kubernetes.io/projected/a784a221-4d26-4f11-b573-cf1046ea769e-kube-api-access-4vcjd\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.006353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.006298 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a784a221-4d26-4f11-b573-cf1046ea769e-manifests\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.006353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.006340 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a784a221-4d26-4f11-b573-cf1046ea769e-tmp\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.006519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.006369 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a784a221-4d26-4f11-b573-cf1046ea769e-webhook-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.006519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.006410 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a784a221-4d26-4f11-b573-cf1046ea769e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.091904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.091873 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:24.107571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.107541 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a784a221-4d26-4f11-b573-cf1046ea769e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.107689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.107638 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a784a221-4d26-4f11-b573-cf1046ea769e-metrics-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.107689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.107669 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcjd\" (UniqueName: \"kubernetes.io/projected/a784a221-4d26-4f11-b573-cf1046ea769e-kube-api-access-4vcjd\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.107802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.107701 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a784a221-4d26-4f11-b573-cf1046ea769e-manifests\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.107918 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.107889 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a784a221-4d26-4f11-b573-cf1046ea769e-tmp\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.108051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.107935 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a784a221-4d26-4f11-b573-cf1046ea769e-webhook-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.108051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.108034 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a784a221-4d26-4f11-b573-cf1046ea769e-manifests\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.109899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.109874 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a784a221-4d26-4f11-b573-cf1046ea769e-tmp\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.109978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.109912 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a784a221-4d26-4f11-b573-cf1046ea769e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.109978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.109921 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a784a221-4d26-4f11-b573-cf1046ea769e-metrics-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.110248 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.110233 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a784a221-4d26-4f11-b573-cf1046ea769e-webhook-cert\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.135711 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.135659 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vcjd\" (UniqueName: \"kubernetes.io/projected/a784a221-4d26-4f11-b573-cf1046ea769e-kube-api-access-4vcjd\") pod \"opendatahub-operator-controller-manager-b54c6978-d2r86\" (UID: \"a784a221-4d26-4f11-b573-cf1046ea769e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.206940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.206917 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:24.331036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.330999 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:24.619785 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.619730 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86"] Oct 30 04:01:24.621748 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:01:24.621717 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda784a221_4d26_4f11_b573_cf1046ea769e.slice/crio-057caebd87f8355366ed32db482b192acf94c536d1cdd07cd063fa3b3e07c361 WatchSource:0}: Error finding container 057caebd87f8355366ed32db482b192acf94c536d1cdd07cd063fa3b3e07c361: Status 404 returned error can't find the container with id 057caebd87f8355366ed32db482b192acf94c536d1cdd07cd063fa3b3e07c361 Oct 30 04:01:24.653876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.653850 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:24.750220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.750188 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5" event={"ID":"e6b32021-61b5-49f5-9af6-6234d5aef6eb","Type":"ContainerStarted","Data":"907f4d9a7f135e20173c0f37d9cf3d36ff2cd95f29acdee60a829fdfc775baa0"} Oct 30 04:01:24.751565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.751535 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" event={"ID":"a784a221-4d26-4f11-b573-cf1046ea769e","Type":"ContainerStarted","Data":"057caebd87f8355366ed32db482b192acf94c536d1cdd07cd063fa3b3e07c361"} Oct 30 04:01:24.803732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.803680 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-d4vw5" podStartSLOduration=2.293730033 podStartE2EDuration="3.803668908s" totalImagesPullingTime="1.509938875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:24.803205657 +0000 UTC m=+208.089080432" watchObservedRunningTime="2026-10-30 04:01:24.803668908 +0000 UTC m=+208.089543682" Oct 30 04:01:24.931597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:24.931567 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:26.302952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:26.302914 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:26.552817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:26.552785 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:27.300679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:27.300640 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:27.766301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:27.766264 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" event={"ID":"a784a221-4d26-4f11-b573-cf1046ea769e","Type":"ContainerStarted","Data":"cc36be1aef6b03a7f407d13acf62d7b1e3632054b0e451bbef8485783ea28fa9"} Oct 30 04:01:28.195122 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:28.195081 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:28.744539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:28.744507 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:28.772125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:28.772098 4793 generic.go:356] "Generic (PLEG): container finished" podID="a784a221-4d26-4f11-b573-cf1046ea769e" containerID="cc36be1aef6b03a7f407d13acf62d7b1e3632054b0e451bbef8485783ea28fa9" exitCode=0 Oct 30 04:01:28.772577 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:28.772140 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" event={"ID":"a784a221-4d26-4f11-b573-cf1046ea769e","Type":"ContainerDied","Data":"cc36be1aef6b03a7f407d13acf62d7b1e3632054b0e451bbef8485783ea28fa9"} Oct 30 04:01:28.981711 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:28.981684 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:28.982361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:28.982340 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d"] Oct 30 04:01:28.984318 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:28.984295 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d" Oct 30 04:01:29.010992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.010935 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d"] Oct 30 04:01:29.072297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.072279 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 04:01:29.072434 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.072319 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 04:01:29.073007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.072993 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5zvn4\"" Oct 30 04:01:29.147675 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.147654 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjsxm\" (UniqueName: \"kubernetes.io/projected/48e2d441-0541-4355-a136-3adf7288eb53-kube-api-access-gjsxm\") pod \"cert-manager-cainjector-5c57bbd4fc-mvz4d\" (UID: \"48e2d441-0541-4355-a136-3adf7288eb53\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d" Oct 30 04:01:29.147777 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.147698 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48e2d441-0541-4355-a136-3adf7288eb53-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mvz4d\" (UID: \"48e2d441-0541-4355-a136-3adf7288eb53\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d" Oct 30 04:01:29.248713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.248684 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48e2d441-0541-4355-a136-3adf7288eb53-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mvz4d\" (UID: \"48e2d441-0541-4355-a136-3adf7288eb53\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d" Oct 30 04:01:29.248842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.248759 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjsxm\" (UniqueName: \"kubernetes.io/projected/48e2d441-0541-4355-a136-3adf7288eb53-kube-api-access-gjsxm\") pod \"cert-manager-cainjector-5c57bbd4fc-mvz4d\" (UID: \"48e2d441-0541-4355-a136-3adf7288eb53\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d" Oct 30 04:01:29.284376 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.284302 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/48e2d441-0541-4355-a136-3adf7288eb53-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mvz4d\" (UID: \"48e2d441-0541-4355-a136-3adf7288eb53\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d" Oct 30 04:01:29.284510 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.284383 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjsxm\" (UniqueName: \"kubernetes.io/projected/48e2d441-0541-4355-a136-3adf7288eb53-kube-api-access-gjsxm\") pod \"cert-manager-cainjector-5c57bbd4fc-mvz4d\" (UID: \"48e2d441-0541-4355-a136-3adf7288eb53\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d" Oct 30 04:01:29.304262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.304239 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d" Oct 30 04:01:29.601454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.601412 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d"] Oct 30 04:01:29.603609 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:01:29.603575 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48e2d441_0541_4355_a136_3adf7288eb53.slice/crio-6e5c0b7eb9a446fe331160423365ca55155e8346f82b66f2a11d32f7d4173dab WatchSource:0}: Error finding container 6e5c0b7eb9a446fe331160423365ca55155e8346f82b66f2a11d32f7d4173dab: Status 404 returned error can't find the container with id 6e5c0b7eb9a446fe331160423365ca55155e8346f82b66f2a11d32f7d4173dab Oct 30 04:01:29.777196 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.777162 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d" event={"ID":"48e2d441-0541-4355-a136-3adf7288eb53","Type":"ContainerStarted","Data":"6e5c0b7eb9a446fe331160423365ca55155e8346f82b66f2a11d32f7d4173dab"} Oct 30 04:01:29.779116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.779091 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" event={"ID":"a784a221-4d26-4f11-b573-cf1046ea769e","Type":"ContainerStarted","Data":"926a9d9dbd363c72067f940d506613eb1fb4fe966f3b672926e7cf7943f5b4f4"} Oct 30 04:01:29.779194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.779180 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:29.847648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:29.847606 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" podStartSLOduration=3.203739035 podStartE2EDuration="6.84759463s" totalImagesPullingTime="3.177634815s" totalInitContainerRuntime="466.22078ms" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:29.846353305 +0000 UTC m=+213.132228080" watchObservedRunningTime="2026-10-30 04:01:29.84759463 +0000 UTC m=+213.133469404" Oct 30 04:01:31.417209 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:31.417175 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg"] Oct 30 04:01:31.419526 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:31.419501 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg" Oct 30 04:01:31.455795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:31.455762 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 04:01:31.455945 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:31.455823 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 04:01:31.455945 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:31.455828 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mrvp9\"" Oct 30 04:01:31.457540 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:31.457521 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg"] Oct 30 04:01:31.568497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:31.568458 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxk88\" (UniqueName: \"kubernetes.io/projected/8af2e200-3999-4f9b-a10c-1407bef727e3-kube-api-access-bxk88\") pod \"authorino-operator-7b7bb4bc77-c4zkg\" (UID: \"8af2e200-3999-4f9b-a10c-1407bef727e3\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg" Oct 30 04:01:31.669843 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:31.669768 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxk88\" (UniqueName: \"kubernetes.io/projected/8af2e200-3999-4f9b-a10c-1407bef727e3-kube-api-access-bxk88\") pod \"authorino-operator-7b7bb4bc77-c4zkg\" (UID: \"8af2e200-3999-4f9b-a10c-1407bef727e3\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg" Oct 30 04:01:31.693657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:31.693632 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxk88\" (UniqueName: \"kubernetes.io/projected/8af2e200-3999-4f9b-a10c-1407bef727e3-kube-api-access-bxk88\") pod \"authorino-operator-7b7bb4bc77-c4zkg\" (UID: \"8af2e200-3999-4f9b-a10c-1407bef727e3\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg" Oct 30 04:01:31.733113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:31.733085 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg" Oct 30 04:01:32.052898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:32.052867 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg"] Oct 30 04:01:32.058609 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:01:32.058580 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af2e200_3999_4f9b_a10c_1407bef727e3.slice/crio-075e2e99197ed35fb2bf12df964332b2bec82ea6871db4ef7b55b7cd9e292520 WatchSource:0}: Error finding container 075e2e99197ed35fb2bf12df964332b2bec82ea6871db4ef7b55b7cd9e292520: Status 404 returned error can't find the container with id 075e2e99197ed35fb2bf12df964332b2bec82ea6871db4ef7b55b7cd9e292520 Oct 30 04:01:32.795256 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:32.795221 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg" event={"ID":"8af2e200-3999-4f9b-a10c-1407bef727e3","Type":"ContainerStarted","Data":"075e2e99197ed35fb2bf12df964332b2bec82ea6871db4ef7b55b7cd9e292520"} Oct 30 04:01:32.796884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:32.796853 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d" event={"ID":"48e2d441-0541-4355-a136-3adf7288eb53","Type":"ContainerStarted","Data":"a8808ecd4172148ae0f6b51391e094b1c02ff83eafce87e7e90b63778bb8d896"} Oct 30 04:01:32.860510 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:32.860454 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mvz4d" podStartSLOduration=2.128448908 podStartE2EDuration="4.860439216s" totalImagesPullingTime="2.731990308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:32.858928933 +0000 UTC m=+216.144803727" watchObservedRunningTime="2026-10-30 04:01:32.860439216 +0000 UTC m=+216.146313991" Oct 30 04:01:33.077625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:33.077518 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:34.808570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:34.808536 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg" event={"ID":"8af2e200-3999-4f9b-a10c-1407bef727e3","Type":"ContainerStarted","Data":"20d6a63684553b8b9986985f79f034e7f2a7fd343817b183235b831256bd47ef"} Oct 30 04:01:34.808969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:34.808618 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg" Oct 30 04:01:34.874771 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:34.874721 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg" podStartSLOduration=1.302932151 podStartE2EDuration="3.874708299s" totalImagesPullingTime="2.571776148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:34.87338411 +0000 UTC m=+218.159258896" watchObservedRunningTime="2026-10-30 04:01:34.874708299 +0000 UTC m=+218.160583073" Oct 30 04:01:35.362711 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:35.362675 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv"] Oct 30 04:01:35.364966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:35.364943 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv" Oct 30 04:01:35.391397 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:35.391374 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv"] Oct 30 04:01:35.400525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:35.400506 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 04:01:35.400639 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:35.400553 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cv8k4\"" Oct 30 04:01:35.500991 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:35.500950 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:35.501205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:35.501184 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pk8z\" (UniqueName: \"kubernetes.io/projected/12b6686c-fc28-4cc0-9c53-252a4e8a02bd-kube-api-access-7pk8z\") pod \"dns-operator-controller-manager-8557cbdb84-pg5vv\" (UID: \"12b6686c-fc28-4cc0-9c53-252a4e8a02bd\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv" Oct 30 04:01:35.602027 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:35.601995 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pk8z\" (UniqueName: \"kubernetes.io/projected/12b6686c-fc28-4cc0-9c53-252a4e8a02bd-kube-api-access-7pk8z\") pod \"dns-operator-controller-manager-8557cbdb84-pg5vv\" (UID: \"12b6686c-fc28-4cc0-9c53-252a4e8a02bd\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv" Oct 30 04:01:35.626159 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:35.626102 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pk8z\" (UniqueName: \"kubernetes.io/projected/12b6686c-fc28-4cc0-9c53-252a4e8a02bd-kube-api-access-7pk8z\") pod \"dns-operator-controller-manager-8557cbdb84-pg5vv\" (UID: \"12b6686c-fc28-4cc0-9c53-252a4e8a02bd\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv" Oct 30 04:01:35.675614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:35.675588 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv" Oct 30 04:01:35.759112 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:35.759084 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:35.967609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:35.967585 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv"] Oct 30 04:01:35.968963 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:01:35.968936 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b6686c_fc28_4cc0_9c53_252a4e8a02bd.slice/crio-cc0595f466f62018c6b361e9b0712fe7a24c247955da2c17f7254dc04f26750c WatchSource:0}: Error finding container cc0595f466f62018c6b361e9b0712fe7a24c247955da2c17f7254dc04f26750c: Status 404 returned error can't find the container with id cc0595f466f62018c6b361e9b0712fe7a24c247955da2c17f7254dc04f26750c Oct 30 04:01:36.051815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:36.051784 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:36.548465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:36.548404 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:36.819004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:36.818919 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv" event={"ID":"12b6686c-fc28-4cc0-9c53-252a4e8a02bd","Type":"ContainerStarted","Data":"cc0595f466f62018c6b361e9b0712fe7a24c247955da2c17f7254dc04f26750c"} Oct 30 04:01:37.018527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:37.018492 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:37.893931 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:37.893896 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:38.158986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:38.158900 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:38.407365 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:38.407321 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:38.669475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:38.669434 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:38.829675 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:38.829641 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv" event={"ID":"12b6686c-fc28-4cc0-9c53-252a4e8a02bd","Type":"ContainerStarted","Data":"18d48df8d1837772f9a933a7b9aa8f1398ec03563f0e2acb4b15d6c4b0902271"} Oct 30 04:01:38.829884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:38.829861 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv" Oct 30 04:01:38.887789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:38.887744 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv" podStartSLOduration=1.958930074 podStartE2EDuration="3.887731903s" totalImagesPullingTime="1.928801829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:38.887080356 +0000 UTC m=+222.172955133" watchObservedRunningTime="2026-10-30 04:01:38.887731903 +0000 UTC m=+222.173606677" Oct 30 04:01:39.007013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:39.006984 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:40.785276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:40.785206 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-b54c6978-d2r86" Oct 30 04:01:41.218202 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:01:41.218170 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:01:41.525299 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.525219 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:41.645967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.645934 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr"] Oct 30 04:01:41.648116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.648086 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr" Oct 30 04:01:41.650459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.650438 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncq9w\" (UniqueName: \"kubernetes.io/projected/036ac5d4-68e5-4129-93ac-b3286b5bc3aa-kube-api-access-ncq9w\") pod \"jobset-operator-674cd59dc5-8lvxr\" (UID: \"036ac5d4-68e5-4129-93ac-b3286b5bc3aa\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr" Oct 30 04:01:41.650553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.650477 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/036ac5d4-68e5-4129-93ac-b3286b5bc3aa-tmp\") pod \"jobset-operator-674cd59dc5-8lvxr\" (UID: \"036ac5d4-68e5-4129-93ac-b3286b5bc3aa\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr" Oct 30 04:01:41.665166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.665147 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 04:01:41.669559 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.669532 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr"] Oct 30 04:01:41.672712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.672692 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-fbx6j\"" Oct 30 04:01:41.741860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.741834 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:41.751849 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.751827 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncq9w\" (UniqueName: \"kubernetes.io/projected/036ac5d4-68e5-4129-93ac-b3286b5bc3aa-kube-api-access-ncq9w\") pod \"jobset-operator-674cd59dc5-8lvxr\" (UID: \"036ac5d4-68e5-4129-93ac-b3286b5bc3aa\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr" Oct 30 04:01:41.751927 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.751864 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/036ac5d4-68e5-4129-93ac-b3286b5bc3aa-tmp\") pod \"jobset-operator-674cd59dc5-8lvxr\" (UID: \"036ac5d4-68e5-4129-93ac-b3286b5bc3aa\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr" Oct 30 04:01:41.752207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.752191 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/036ac5d4-68e5-4129-93ac-b3286b5bc3aa-tmp\") pod \"jobset-operator-674cd59dc5-8lvxr\" (UID: \"036ac5d4-68e5-4129-93ac-b3286b5bc3aa\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr" Oct 30 04:01:41.764262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.764243 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 04:01:41.780718 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.780666 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncq9w\" (UniqueName: \"kubernetes.io/projected/036ac5d4-68e5-4129-93ac-b3286b5bc3aa-kube-api-access-ncq9w\") pod \"jobset-operator-674cd59dc5-8lvxr\" (UID: \"036ac5d4-68e5-4129-93ac-b3286b5bc3aa\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr" Oct 30 04:01:41.958242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:41.958217 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr" Oct 30 04:01:42.252251 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:42.252228 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr"] Oct 30 04:01:42.253815 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:01:42.253787 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod036ac5d4_68e5_4129_93ac_b3286b5bc3aa.slice/crio-40769a04b75910ea1d78e221a7c1525efe984cfaed80830edc08764d65278561 WatchSource:0}: Error finding container 40769a04b75910ea1d78e221a7c1525efe984cfaed80830edc08764d65278561: Status 404 returned error can't find the container with id 40769a04b75910ea1d78e221a7c1525efe984cfaed80830edc08764d65278561 Oct 30 04:01:42.846293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:42.846258 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr" event={"ID":"036ac5d4-68e5-4129-93ac-b3286b5bc3aa","Type":"ContainerStarted","Data":"40769a04b75910ea1d78e221a7c1525efe984cfaed80830edc08764d65278561"} Oct 30 04:01:43.730928 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:43.730898 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q"] Oct 30 04:01:43.733689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:43.733672 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" Oct 30 04:01:43.749521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:43.749495 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q"] Oct 30 04:01:43.762274 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:43.762252 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wmr7g\"" Oct 30 04:01:43.763954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:43.763936 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/961819a6-71ec-43e0-bd1d-976e5e55eada-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7pv2q\" (UID: \"961819a6-71ec-43e0-bd1d-976e5e55eada\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" Oct 30 04:01:43.764005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:43.763984 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h79x\" (UniqueName: \"kubernetes.io/projected/961819a6-71ec-43e0-bd1d-976e5e55eada-kube-api-access-9h79x\") pod \"kuadrant-operator-controller-manager-678d87766f-7pv2q\" (UID: \"961819a6-71ec-43e0-bd1d-976e5e55eada\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" Oct 30 04:01:43.864883 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:43.864852 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/961819a6-71ec-43e0-bd1d-976e5e55eada-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7pv2q\" (UID: \"961819a6-71ec-43e0-bd1d-976e5e55eada\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" Oct 30 04:01:43.865036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:43.864896 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h79x\" (UniqueName: \"kubernetes.io/projected/961819a6-71ec-43e0-bd1d-976e5e55eada-kube-api-access-9h79x\") pod \"kuadrant-operator-controller-manager-678d87766f-7pv2q\" (UID: \"961819a6-71ec-43e0-bd1d-976e5e55eada\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" Oct 30 04:01:43.865271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:43.865247 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/961819a6-71ec-43e0-bd1d-976e5e55eada-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7pv2q\" (UID: \"961819a6-71ec-43e0-bd1d-976e5e55eada\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" Oct 30 04:01:43.888930 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:43.888903 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h79x\" (UniqueName: \"kubernetes.io/projected/961819a6-71ec-43e0-bd1d-976e5e55eada-kube-api-access-9h79x\") pod \"kuadrant-operator-controller-manager-678d87766f-7pv2q\" (UID: \"961819a6-71ec-43e0-bd1d-976e5e55eada\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" Oct 30 04:01:44.045298 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:44.045239 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" Oct 30 04:01:44.364625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:44.364554 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q"] Oct 30 04:01:44.377618 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:01:44.377580 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961819a6_71ec_43e0_bd1d_976e5e55eada.slice/crio-c70b60af832035c0c58f3d7d1b0fa605678631ad8765bc8f26407629eb1df96b WatchSource:0}: Error finding container c70b60af832035c0c58f3d7d1b0fa605678631ad8765bc8f26407629eb1df96b: Status 404 returned error can't find the container with id c70b60af832035c0c58f3d7d1b0fa605678631ad8765bc8f26407629eb1df96b Oct 30 04:01:44.839793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:44.839756 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:44.855965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:44.855937 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" event={"ID":"961819a6-71ec-43e0-bd1d-976e5e55eada","Type":"ContainerStarted","Data":"c70b60af832035c0c58f3d7d1b0fa605678631ad8765bc8f26407629eb1df96b"} Oct 30 04:01:45.815859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:45.815819 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-c4zkg" Oct 30 04:01:46.867158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:46.867123 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr" event={"ID":"036ac5d4-68e5-4129-93ac-b3286b5bc3aa","Type":"ContainerStarted","Data":"44b7de8ee98ed64c9f312b4c4f47b71d3366118a3645feb4dc6bc7e6c29f7e23"} Oct 30 04:01:46.912989 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:46.912928 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-8lvxr" podStartSLOduration=2.080831478 podStartE2EDuration="5.912910874s" totalImagesPullingTime="3.832079396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:46.911128616 +0000 UTC m=+230.197003394" watchObservedRunningTime="2026-10-30 04:01:46.912910874 +0000 UTC m=+230.198785652" Oct 30 04:01:48.684335 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:48.684299 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:48.878852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:48.878818 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" event={"ID":"961819a6-71ec-43e0-bd1d-976e5e55eada","Type":"ContainerStarted","Data":"8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4"} Oct 30 04:01:48.879012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:48.878956 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" Oct 30 04:01:48.896887 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:48.896858 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:48.951711 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:48.951633 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" podStartSLOduration=1.998805639 podStartE2EDuration="5.951621808s" totalImagesPullingTime="3.952816169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:48.937676369 +0000 UTC m=+232.223551144" watchObservedRunningTime="2026-10-30 04:01:48.951621808 +0000 UTC m=+232.237496583" Oct 30 04:01:49.633164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:49.633121 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:49.836551 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:49.836523 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-pg5vv" Oct 30 04:01:49.900963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:49.900889 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:50.140364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:50.140328 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:50.365372 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:50.365335 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:53.423234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:53.423196 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:53.642854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:53.642814 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:58.898982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:58.898946 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:59.122371 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.122340 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:59.344748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.344708 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:59.672004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.671928 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:59.820264 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.820230 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2"] Oct 30 04:01:59.880758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.880718 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2"] Oct 30 04:01:59.880908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.880833 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:01:59.884696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.884674 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" Oct 30 04:01:59.895788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.895769 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jtbjb\"" Oct 30 04:01:59.934041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.934014 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:01:59.985596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.985570 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:01:59.985758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.985659 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:01:59.985758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.985701 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:01:59.985758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.985732 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:01:59.985758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.985754 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:01:59.986013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.985810 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:01:59.986013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.985836 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28tqx\" (UniqueName: \"kubernetes.io/projected/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-kube-api-access-28tqx\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:01:59.986013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.985877 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:01:59.986013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:01:59.985926 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.086309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.086277 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.086309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.086314 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28tqx\" (UniqueName: \"kubernetes.io/projected/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-kube-api-access-28tqx\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.086573 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.086350 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.086573 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.086392 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.086573 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.086483 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.086893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.086867 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.086946 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.086670 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.086992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.086963 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.087033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.086973 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.087033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.087008 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.087105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.087031 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.087105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.087061 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.087105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.087089 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.087276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.087251 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.089113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.089091 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.089528 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.089509 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.112118 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.112094 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.112915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.112888 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28tqx\" (UniqueName: \"kubernetes.io/projected/1179efa6-bb34-43ae-ab5d-ae9e3ec68028-kube-api-access-28tqx\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2\" (UID: \"1179efa6-bb34-43ae-ab5d-ae9e3ec68028\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.157358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.157324 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:00.206889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.206817 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:00.391149 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.391086 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:00.580594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.580463 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2"] Oct 30 04:02:00.583056 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:02:00.583026 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1179efa6_bb34_43ae_ab5d_ae9e3ec68028.slice/crio-9f9592ed19301a37078278be0d4120a4c4513fbcb876e076b2e74aed045cda58 WatchSource:0}: Error finding container 9f9592ed19301a37078278be0d4120a4c4513fbcb876e076b2e74aed045cda58: Status 404 returned error can't find the container with id 9f9592ed19301a37078278be0d4120a4c4513fbcb876e076b2e74aed045cda58 Oct 30 04:02:00.585334 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.585303 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:02:00.585398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.585370 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:02:00.585460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.585396 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:02:00.651411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.651380 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:00.931446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.931382 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" event={"ID":"1179efa6-bb34-43ae-ab5d-ae9e3ec68028","Type":"ContainerStarted","Data":"f3ae4e0b4d70f87b158c196696bae0e008f1152fc888ebe309b1dbeec89282aa"} Oct 30 04:02:00.931626 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.931458 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" event={"ID":"1179efa6-bb34-43ae-ab5d-ae9e3ec68028","Type":"ContainerStarted","Data":"9f9592ed19301a37078278be0d4120a4c4513fbcb876e076b2e74aed045cda58"} Oct 30 04:02:00.980228 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:00.980176 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" podStartSLOduration=1.980163819 podStartE2EDuration="1.980163819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:00.979400048 +0000 UTC m=+244.265274824" watchObservedRunningTime="2026-10-30 04:02:00.980163819 +0000 UTC m=+244.266038594" Oct 30 04:02:01.138973 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:01.138938 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:01.207499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:01.207404 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:01.211931 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:01.211907 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:01.378900 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:01.378867 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:01.936278 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:01.936248 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:01.937326 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:01.937300 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2" Oct 30 04:02:02.656070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:02.656036 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:02.869493 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:02.869452 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:07.300009 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:07.299974 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:07.513168 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:07.513130 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:08.173821 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.173791 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:08.447134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.447100 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:08.790442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.790311 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm"] Oct 30 04:02:08.795467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.795443 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:08.812047 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.812028 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 04:02:08.824624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.824604 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-zvzr6\"" Oct 30 04:02:08.824832 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.824809 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 04:02:08.824832 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.824822 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 04:02:08.826468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.826449 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm"] Oct 30 04:02:08.865771 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.865745 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/118752b7-2596-455e-ba27-8369b7ba3cd8-cert\") pod \"jobset-controller-manager-5df7965469-nxccm\" (UID: \"118752b7-2596-455e-ba27-8369b7ba3cd8\") " pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:08.865882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.865782 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/118752b7-2596-455e-ba27-8369b7ba3cd8-manager-config\") pod \"jobset-controller-manager-5df7965469-nxccm\" (UID: \"118752b7-2596-455e-ba27-8369b7ba3cd8\") " pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:08.865882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.865813 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/118752b7-2596-455e-ba27-8369b7ba3cd8-metrics-certs\") pod \"jobset-controller-manager-5df7965469-nxccm\" (UID: \"118752b7-2596-455e-ba27-8369b7ba3cd8\") " pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:08.865954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.865923 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68bbp\" (UniqueName: \"kubernetes.io/projected/118752b7-2596-455e-ba27-8369b7ba3cd8-kube-api-access-68bbp\") pod \"jobset-controller-manager-5df7965469-nxccm\" (UID: \"118752b7-2596-455e-ba27-8369b7ba3cd8\") " pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:08.966276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.966246 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/118752b7-2596-455e-ba27-8369b7ba3cd8-manager-config\") pod \"jobset-controller-manager-5df7965469-nxccm\" (UID: \"118752b7-2596-455e-ba27-8369b7ba3cd8\") " pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:08.966450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.966286 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/118752b7-2596-455e-ba27-8369b7ba3cd8-metrics-certs\") pod \"jobset-controller-manager-5df7965469-nxccm\" (UID: \"118752b7-2596-455e-ba27-8369b7ba3cd8\") " pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:08.966527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.966458 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68bbp\" (UniqueName: \"kubernetes.io/projected/118752b7-2596-455e-ba27-8369b7ba3cd8-kube-api-access-68bbp\") pod \"jobset-controller-manager-5df7965469-nxccm\" (UID: \"118752b7-2596-455e-ba27-8369b7ba3cd8\") " pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:08.966627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.966610 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/118752b7-2596-455e-ba27-8369b7ba3cd8-cert\") pod \"jobset-controller-manager-5df7965469-nxccm\" (UID: \"118752b7-2596-455e-ba27-8369b7ba3cd8\") " pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:08.967044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.967019 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/118752b7-2596-455e-ba27-8369b7ba3cd8-manager-config\") pod \"jobset-controller-manager-5df7965469-nxccm\" (UID: \"118752b7-2596-455e-ba27-8369b7ba3cd8\") " pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:08.969108 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.969085 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/118752b7-2596-455e-ba27-8369b7ba3cd8-metrics-certs\") pod \"jobset-controller-manager-5df7965469-nxccm\" (UID: \"118752b7-2596-455e-ba27-8369b7ba3cd8\") " pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:08.969202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.969085 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/118752b7-2596-455e-ba27-8369b7ba3cd8-cert\") pod \"jobset-controller-manager-5df7965469-nxccm\" (UID: \"118752b7-2596-455e-ba27-8369b7ba3cd8\") " pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:08.989949 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:08.989919 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68bbp\" (UniqueName: \"kubernetes.io/projected/118752b7-2596-455e-ba27-8369b7ba3cd8-kube-api-access-68bbp\") pod \"jobset-controller-manager-5df7965469-nxccm\" (UID: \"118752b7-2596-455e-ba27-8369b7ba3cd8\") " pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:09.106723 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:09.106642 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:09.400571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:09.400547 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm"] Oct 30 04:02:09.402293 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:02:09.402263 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118752b7_2596_455e_ba27_8369b7ba3cd8.slice/crio-b611bb3df78b3ab52c9d9801604dfe894aa6d6c8e1b6a33530218d7e0e3c200f WatchSource:0}: Error finding container b611bb3df78b3ab52c9d9801604dfe894aa6d6c8e1b6a33530218d7e0e3c200f: Status 404 returned error can't find the container with id b611bb3df78b3ab52c9d9801604dfe894aa6d6c8e1b6a33530218d7e0e3c200f Oct 30 04:02:09.973334 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:09.973301 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" event={"ID":"118752b7-2596-455e-ba27-8369b7ba3cd8","Type":"ContainerStarted","Data":"b611bb3df78b3ab52c9d9801604dfe894aa6d6c8e1b6a33530218d7e0e3c200f"} Oct 30 04:02:09.997073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:09.997042 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:10.235269 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:10.235181 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:11.224924 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:02:11.224875 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:02:12.633308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:12.633276 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:12.851572 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:12.851485 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:12.993426 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:12.993370 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" event={"ID":"118752b7-2596-455e-ba27-8369b7ba3cd8","Type":"ContainerStarted","Data":"40c450f5bf80c085da7012c60f612810d37bddec59719cd20cee61cf41e18894"} Oct 30 04:02:12.993590 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:12.993565 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:13.035842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:13.035800 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" podStartSLOduration=1.907366087 podStartE2EDuration="5.03578684s" totalImagesPullingTime="3.128420753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:02:13.03457583 +0000 UTC m=+256.320450604" watchObservedRunningTime="2026-10-30 04:02:13.03578684 +0000 UTC m=+256.321661614" Oct 30 04:02:19.034909 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:19.034869 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:19.247522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:19.247476 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:24.003090 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:24.003056 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5df7965469-nxccm" Oct 30 04:02:24.810389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:24.810353 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:25.040388 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:25.040358 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:26.090403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:26.090358 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:26.301933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:26.301898 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:27.757787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:27.757757 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:27.968755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:27.968721 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:29.046900 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:29.046862 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:29.266684 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:29.266647 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:32.527185 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:32.527138 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:32.740349 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:32.740315 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:38.286265 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:38.286229 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:38.497849 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:38.497813 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:38.711584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:38.711551 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:41.218019 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:02:41.217981 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:02:45.335221 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:45.335180 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:45.544536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:45.544505 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:50.318797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:50.318752 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:59.433061 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.433029 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:59.731740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.731667 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:59.829353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.829323 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-pxxpk"] Oct 30 04:02:59.833973 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.833950 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:02:59.853378 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.853354 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-pxxpk"] Oct 30 04:02:59.853518 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.853506 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 04:02:59.853584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.853530 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 04:02:59.866910 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.866883 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66328127-fc0b-4f8a-9599-2a82c297712d-cert\") pod \"odh-model-controller-769875db56-pxxpk\" (UID: \"66328127-fc0b-4f8a-9599-2a82c297712d\") " pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:02:59.867016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.866932 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66328127-fc0b-4f8a-9599-2a82c297712d-metrics-certs\") pod \"odh-model-controller-769875db56-pxxpk\" (UID: \"66328127-fc0b-4f8a-9599-2a82c297712d\") " pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:02:59.867055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.867009 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpjsh\" (UniqueName: \"kubernetes.io/projected/66328127-fc0b-4f8a-9599-2a82c297712d-kube-api-access-vpjsh\") pod \"odh-model-controller-769875db56-pxxpk\" (UID: \"66328127-fc0b-4f8a-9599-2a82c297712d\") " pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:02:59.910956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.910935 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-tj8md\"" Oct 30 04:02:59.910956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.910946 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 04:02:59.911110 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.910971 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 04:02:59.960718 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.960689 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:02:59.967704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.967676 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66328127-fc0b-4f8a-9599-2a82c297712d-cert\") pod \"odh-model-controller-769875db56-pxxpk\" (UID: \"66328127-fc0b-4f8a-9599-2a82c297712d\") " pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:02:59.967801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.967724 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66328127-fc0b-4f8a-9599-2a82c297712d-metrics-certs\") pod \"odh-model-controller-769875db56-pxxpk\" (UID: \"66328127-fc0b-4f8a-9599-2a82c297712d\") " pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:02:59.967801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.967753 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpjsh\" (UniqueName: \"kubernetes.io/projected/66328127-fc0b-4f8a-9599-2a82c297712d-kube-api-access-vpjsh\") pod \"odh-model-controller-769875db56-pxxpk\" (UID: \"66328127-fc0b-4f8a-9599-2a82c297712d\") " pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:02:59.970042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.970021 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66328127-fc0b-4f8a-9599-2a82c297712d-metrics-certs\") pod \"odh-model-controller-769875db56-pxxpk\" (UID: \"66328127-fc0b-4f8a-9599-2a82c297712d\") " pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:02:59.970127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.970072 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66328127-fc0b-4f8a-9599-2a82c297712d-cert\") pod \"odh-model-controller-769875db56-pxxpk\" (UID: \"66328127-fc0b-4f8a-9599-2a82c297712d\") " pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:02:59.990809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:02:59.990764 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpjsh\" (UniqueName: \"kubernetes.io/projected/66328127-fc0b-4f8a-9599-2a82c297712d-kube-api-access-vpjsh\") pod \"odh-model-controller-769875db56-pxxpk\" (UID: \"66328127-fc0b-4f8a-9599-2a82c297712d\") " pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:03:00.146089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:00.146063 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:03:00.183853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:00.183813 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:00.401915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:00.401833 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:00.438458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:00.438432 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-pxxpk"] Oct 30 04:03:00.439721 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:03:00.439690 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66328127_fc0b_4f8a_9599_2a82c297712d.slice/crio-9f5be5a393bcd1a555d648b389da5a9ae4e51f0143c21a1d60a6047b815e4766 WatchSource:0}: Error finding container 9f5be5a393bcd1a555d648b389da5a9ae4e51f0143c21a1d60a6047b815e4766: Status 404 returned error can't find the container with id 9f5be5a393bcd1a555d648b389da5a9ae4e51f0143c21a1d60a6047b815e4766 Oct 30 04:03:01.221867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:01.221824 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-pxxpk" event={"ID":"66328127-fc0b-4f8a-9599-2a82c297712d","Type":"ContainerStarted","Data":"9f5be5a393bcd1a555d648b389da5a9ae4e51f0143c21a1d60a6047b815e4766"} Oct 30 04:03:02.190601 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:02.190563 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:02.419219 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:02.419176 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:02.635158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:02.635062 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:03:03.234540 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:03.234453 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-pxxpk" event={"ID":"66328127-fc0b-4f8a-9599-2a82c297712d","Type":"ContainerStarted","Data":"8b58ddc47a898e23abda6f1f0d8c2e0067b01dae8164f7b1794818f6691a4fa0"} Oct 30 04:03:03.234891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:03.234559 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:03:03.277925 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:03.277865 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-pxxpk" podStartSLOduration=1.778336114 podStartE2EDuration="4.277854798s" totalImagesPullingTime="2.499518684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:02:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:03:03.276992964 +0000 UTC m=+306.562867739" watchObservedRunningTime="2026-10-30 04:03:03.277854798 +0000 UTC m=+306.563729619" Oct 30 04:03:11.217790 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:03:11.217709 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:03:14.241497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:14.241469 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-pxxpk" Oct 30 04:03:41.216623 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:03:41.216587 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:03:50.436480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:03:50.436413 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:11.217852 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:04:11.217811 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:04:39.296688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:04:39.296654 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:39.515446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:04:39.515349 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:04:41.218502 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:04:41.218469 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:04:50.571249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:04:50.571216 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:01.019547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:05:01.019509 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:01.235143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:05:01.235113 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:06.028017 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:05:06.027983 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:07.245311 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:05:07.245273 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:11.218628 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:05:11.218587 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:05:11.580121 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:05:11.580023 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:11.789692 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:05:11.789659 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:28.291383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:05:28.291348 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:05:41.216701 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:05:41.216667 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:05:50.708677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:05:50.708645 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:11.218744 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:06:11.218668 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:06:25.706405 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:06:25.706369 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:27.545884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:06:27.545848 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:06:41.217131 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:06:41.217093 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:06:50.843712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:06:50.843676 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:07:11.217261 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:07:11.217225 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:07:41.220033 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:07:41.217842 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:07:50.932707 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:07:50.932666 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:08:11.216830 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:08:11.216793 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:08:41.216963 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:08:41.216924 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:08:51.120617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:08:51.120584 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:09:11.217307 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:09:11.217227 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:09:41.217827 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:09:41.217789 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:09:51.212121 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:09:51.212072 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:10:11.217436 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:10:11.217383 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:10:41.217893 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:10:41.217812 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:10:51.284369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:10:51.284328 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:11:11.217146 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:11:11.217107 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:11:41.217613 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:11:41.217576 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:11:51.485367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:11:51.485330 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:11.217564 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:12:11.217486 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:12:19.853285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:19.853252 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7pv2q_961819a6-71ec-43e0-bd1d-976e5e55eada/manager/0.log" Oct 30 04:12:20.096067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.096034 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q"] Oct 30 04:12:20.096315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.096293 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" podUID="961819a6-71ec-43e0-bd1d-976e5e55eada" containerName="manager" containerID="cri-o://8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4" gracePeriod=10 Oct 30 04:12:20.125358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.125335 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg"] Oct 30 04:12:20.127880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.127864 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" Oct 30 04:12:20.159962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.159934 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg"] Oct 30 04:12:20.241170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.241137 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkqc\" (UniqueName: \"kubernetes.io/projected/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-kube-api-access-zhkqc\") pod \"kuadrant-operator-controller-manager-678d87766f-tcmbg\" (UID: \"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" Oct 30 04:12:20.241331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.241178 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tcmbg\" (UID: \"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" Oct 30 04:12:20.330846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.330822 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" Oct 30 04:12:20.342060 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.342039 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkqc\" (UniqueName: \"kubernetes.io/projected/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-kube-api-access-zhkqc\") pod \"kuadrant-operator-controller-manager-678d87766f-tcmbg\" (UID: \"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" Oct 30 04:12:20.342132 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.342072 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tcmbg\" (UID: \"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" Oct 30 04:12:20.342460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.342443 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tcmbg\" (UID: \"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" Oct 30 04:12:20.365811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.365761 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkqc\" (UniqueName: \"kubernetes.io/projected/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-kube-api-access-zhkqc\") pod \"kuadrant-operator-controller-manager-678d87766f-tcmbg\" (UID: \"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" Oct 30 04:12:20.443053 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.443021 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961819a6-71ec-43e0-bd1d-976e5e55eada-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/961819a6-71ec-43e0-bd1d-976e5e55eada-extensions-socket-volume\") pod \"961819a6-71ec-43e0-bd1d-976e5e55eada\" (UID: \"961819a6-71ec-43e0-bd1d-976e5e55eada\") " Oct 30 04:12:20.443053 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.443056 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961819a6-71ec-43e0-bd1d-976e5e55eada-kube-api-access-9h79x\" (UniqueName: \"kubernetes.io/projected/961819a6-71ec-43e0-bd1d-976e5e55eada-kube-api-access-9h79x\") pod \"961819a6-71ec-43e0-bd1d-976e5e55eada\" (UID: \"961819a6-71ec-43e0-bd1d-976e5e55eada\") " Oct 30 04:12:20.443487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.443453 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961819a6-71ec-43e0-bd1d-976e5e55eada-extensions-socket-volume" pod "961819a6-71ec-43e0-bd1d-976e5e55eada" (UID: "961819a6-71ec-43e0-bd1d-976e5e55eada"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:12:20.445130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.445105 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961819a6-71ec-43e0-bd1d-976e5e55eada-kube-api-access-9h79x" pod "961819a6-71ec-43e0-bd1d-976e5e55eada" (UID: "961819a6-71ec-43e0-bd1d-976e5e55eada"). InnerVolumeSpecName "kube-api-access-9h79x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:12:20.469281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.469258 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" Oct 30 04:12:20.543829 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.543801 4793 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/961819a6-71ec-43e0-bd1d-976e5e55eada-extensions-socket-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:12:20.543829 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.543827 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h79x\" (UniqueName: \"kubernetes.io/projected/961819a6-71ec-43e0-bd1d-976e5e55eada-kube-api-access-9h79x\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:12:20.702585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.702557 4793 generic.go:356] "Generic (PLEG): container finished" podID="961819a6-71ec-43e0-bd1d-976e5e55eada" containerID="8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4" exitCode=0 Oct 30 04:12:20.702746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.702611 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" Oct 30 04:12:20.702746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.702626 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" event={"ID":"961819a6-71ec-43e0-bd1d-976e5e55eada","Type":"ContainerDied","Data":"8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4"} Oct 30 04:12:20.702746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.702669 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q" event={"ID":"961819a6-71ec-43e0-bd1d-976e5e55eada","Type":"ContainerDied","Data":"c70b60af832035c0c58f3d7d1b0fa605678631ad8765bc8f26407629eb1df96b"} Oct 30 04:12:20.702746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.702693 4793 scope.go:122] "RemoveContainer" containerID="8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4" Oct 30 04:12:20.713263 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.713241 4793 scope.go:122] "RemoveContainer" containerID="8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4" Oct 30 04:12:20.713718 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:12:20.713693 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4\": container with ID starting with 8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4 not found: ID does not exist" containerID="8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4" Oct 30 04:12:20.713801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.713727 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4"} err="failed to get container status \"8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4\": rpc error: code = NotFound desc = could not find container \"8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4\": container with ID starting with 8bf4c7b110edad0910af55e8ebbaa50540464f4a9098ec71aeba01a246b397c4 not found: ID does not exist" Oct 30 04:12:20.758996 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.758971 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q"] Oct 30 04:12:20.763292 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.763273 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7pv2q"] Oct 30 04:12:20.786135 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.786116 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg"] Oct 30 04:12:20.787272 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:12:20.787253 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7da71d_ab0a_44a7_bcc6_b0dd09f7f793.slice/crio-af056a9df554968938b268cfd9794f45e64b8ba4df9390c814824cec4c22a0a5 WatchSource:0}: Error finding container af056a9df554968938b268cfd9794f45e64b8ba4df9390c814824cec4c22a0a5: Status 404 returned error can't find the container with id af056a9df554968938b268cfd9794f45e64b8ba4df9390c814824cec4c22a0a5 Oct 30 04:12:20.844021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:20.843999 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961819a6-71ec-43e0-bd1d-976e5e55eada" path="/var/lib/kubelet/pods/961819a6-71ec-43e0-bd1d-976e5e55eada/volumes" Oct 30 04:12:21.709592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:21.709559 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" event={"ID":"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793","Type":"ContainerStarted","Data":"488d38c78ed8fe2f4e6f00a62f67b44c23fcccfc1093543e4475c9c97d9be3fe"} Oct 30 04:12:21.709940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:21.709635 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" event={"ID":"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793","Type":"ContainerStarted","Data":"af056a9df554968938b268cfd9794f45e64b8ba4df9390c814824cec4c22a0a5"} Oct 30 04:12:21.709940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:21.709673 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" Oct 30 04:12:21.783102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:21.783053 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" podStartSLOduration=1.783039156 podStartE2EDuration="1.783039156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:21.769554308 +0000 UTC m=+865.055429074" watchObservedRunningTime="2026-10-30 04:12:21.783039156 +0000 UTC m=+865.068913930" Oct 30 04:12:32.715833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:32.715801 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" Oct 30 04:12:38.063695 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:38.063653 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:38.276710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:38.276671 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:38.483056 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:38.483020 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:39.088672 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.088626 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:39.297198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.297160 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:39.595469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.595435 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m6qxl"] Oct 30 04:12:39.595814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.595797 4793 cpu_manager.go:414] "containerMap: removing container" podUID="961819a6-71ec-43e0-bd1d-976e5e55eada" containerName="manager" Oct 30 04:12:39.595914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.595815 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961819a6-71ec-43e0-bd1d-976e5e55eada" containerName="manager" Oct 30 04:12:39.595914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.595880 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="961819a6-71ec-43e0-bd1d-976e5e55eada" containerName="manager" Oct 30 04:12:39.598691 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.598672 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" Oct 30 04:12:39.620531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.620507 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m6qxl"] Oct 30 04:12:39.631234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.631205 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rzxvm\"" Oct 30 04:12:39.692458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.692433 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbd57\" (UniqueName: \"kubernetes.io/projected/a0f7828b-c780-4dad-80bb-c42935eafee1-kube-api-access-vbd57\") pod \"authorino-56c6c9ddd5-m6qxl\" (UID: \"a0f7828b-c780-4dad-80bb-c42935eafee1\") " pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" Oct 30 04:12:39.793294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.793270 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbd57\" (UniqueName: \"kubernetes.io/projected/a0f7828b-c780-4dad-80bb-c42935eafee1-kube-api-access-vbd57\") pod \"authorino-56c6c9ddd5-m6qxl\" (UID: \"a0f7828b-c780-4dad-80bb-c42935eafee1\") " pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" Oct 30 04:12:39.816892 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.816861 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbd57\" (UniqueName: \"kubernetes.io/projected/a0f7828b-c780-4dad-80bb-c42935eafee1-kube-api-access-vbd57\") pod \"authorino-56c6c9ddd5-m6qxl\" (UID: \"a0f7828b-c780-4dad-80bb-c42935eafee1\") " pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" Oct 30 04:12:39.909562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:39.909506 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" Oct 30 04:12:40.198912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:40.198883 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m6qxl"] Oct 30 04:12:40.200290 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:12:40.200262 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0f7828b_c780_4dad_80bb_c42935eafee1.slice/crio-1a4f38ffc164620600dcef2d84106740d94553b31e4ceffa72624b1c5c8daff8 WatchSource:0}: Error finding container 1a4f38ffc164620600dcef2d84106740d94553b31e4ceffa72624b1c5c8daff8: Status 404 returned error can't find the container with id 1a4f38ffc164620600dcef2d84106740d94553b31e4ceffa72624b1c5c8daff8 Oct 30 04:12:40.201728 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:40.201708 4793 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:12:40.795534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:40.795502 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" event={"ID":"a0f7828b-c780-4dad-80bb-c42935eafee1","Type":"ContainerStarted","Data":"1a4f38ffc164620600dcef2d84106740d94553b31e4ceffa72624b1c5c8daff8"} Oct 30 04:12:41.221813 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:12:41.221709 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:12:41.809605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:41.809566 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" event={"ID":"a0f7828b-c780-4dad-80bb-c42935eafee1","Type":"ContainerStarted","Data":"dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4"} Oct 30 04:12:41.849633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:41.849583 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" podStartSLOduration=1.450183221 podStartE2EDuration="2.849569874s" totalImagesPullingTime="1.399386653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:12:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:12:41.848108291 +0000 UTC m=+885.133983066" watchObservedRunningTime="2026-10-30 04:12:41.849569874 +0000 UTC m=+885.135444649" Oct 30 04:12:51.533286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:51.533251 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:56.064957 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:56.064922 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:12:56.370778 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:12:56.370697 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:11.186780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:13:11.186744 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:11.217343 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:13:11.217314 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:13:30.408482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:13:30.408448 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:30.682733 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:13:30.682701 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:30.896941 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:13:30.896905 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:34.189089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:13:34.189058 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:34.449748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:13:34.449670 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:36.253076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:13:36.253036 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:36.480295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:13:36.480257 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:41.217000 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:13:41.216919 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:13:45.323906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:13:45.323877 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:13:51.662686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:13:51.662645 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:02.661399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:02.661357 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:02.908846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:02.908815 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:04.288125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:04.288088 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:04.526912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:04.526879 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:11.218764 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:14:11.218730 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:14:14.776682 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:14.776650 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:14.981247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:14.981208 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:14.986015 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:14.985990 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-57jln"] Oct 30 04:14:14.988099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:14.988078 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-57jln" Oct 30 04:14:15.005939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.005914 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-57jln"] Oct 30 04:14:15.146725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.146644 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rz45\" (UniqueName: \"kubernetes.io/projected/1b0b2dc7-d28e-4ef4-8dcb-c4467b272089-kube-api-access-2rz45\") pod \"authorino-6f6944bcdd-57jln\" (UID: \"1b0b2dc7-d28e-4ef4-8dcb-c4467b272089\") " pod="kuadrant-system/authorino-6f6944bcdd-57jln" Oct 30 04:14:15.155204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.155171 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-57jln"] Oct 30 04:14:15.155858 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:14:15.155817 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2rz45], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-57jln" podUID="1b0b2dc7-d28e-4ef4-8dcb-c4467b272089" Oct 30 04:14:15.182444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.182400 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nvhp8"] Oct 30 04:14:15.184799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.184769 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-nvhp8" Oct 30 04:14:15.205787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.205761 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nvhp8"] Oct 30 04:14:15.206029 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.205999 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 04:14:15.225863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.225834 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-57jln" Oct 30 04:14:15.231990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.231972 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-57jln" Oct 30 04:14:15.247830 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.247810 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rz45\" (UniqueName: \"kubernetes.io/projected/1b0b2dc7-d28e-4ef4-8dcb-c4467b272089-kube-api-access-2rz45\") pod \"authorino-6f6944bcdd-57jln\" (UID: \"1b0b2dc7-d28e-4ef4-8dcb-c4467b272089\") " pod="kuadrant-system/authorino-6f6944bcdd-57jln" Oct 30 04:14:15.272227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.272193 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rz45\" (UniqueName: \"kubernetes.io/projected/1b0b2dc7-d28e-4ef4-8dcb-c4467b272089-kube-api-access-2rz45\") pod \"authorino-6f6944bcdd-57jln\" (UID: \"1b0b2dc7-d28e-4ef4-8dcb-c4467b272089\") " pod="kuadrant-system/authorino-6f6944bcdd-57jln" Oct 30 04:14:15.348474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.348408 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpzsp\" (UniqueName: \"kubernetes.io/projected/9ceee7d0-aaea-465f-854e-4d883cad4bd5-kube-api-access-bpzsp\") pod \"authorino-6889cbd8c4-nvhp8\" (UID: \"9ceee7d0-aaea-465f-854e-4d883cad4bd5\") " pod="kuadrant-system/authorino-6889cbd8c4-nvhp8" Oct 30 04:14:15.348615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.348556 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ceee7d0-aaea-465f-854e-4d883cad4bd5-tls-cert\") pod \"authorino-6889cbd8c4-nvhp8\" (UID: \"9ceee7d0-aaea-465f-854e-4d883cad4bd5\") " pod="kuadrant-system/authorino-6889cbd8c4-nvhp8" Oct 30 04:14:15.394084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.394055 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nvhp8"] Oct 30 04:14:15.394243 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:14:15.394224 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bpzsp tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-nvhp8" podUID="9ceee7d0-aaea-465f-854e-4d883cad4bd5" Oct 30 04:14:15.449565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.449540 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0b2dc7-d28e-4ef4-8dcb-c4467b272089-kube-api-access-2rz45\" (UniqueName: \"kubernetes.io/projected/1b0b2dc7-d28e-4ef4-8dcb-c4467b272089-kube-api-access-2rz45\") pod \"1b0b2dc7-d28e-4ef4-8dcb-c4467b272089\" (UID: \"1b0b2dc7-d28e-4ef4-8dcb-c4467b272089\") " Oct 30 04:14:15.449701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.449671 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpzsp\" (UniqueName: \"kubernetes.io/projected/9ceee7d0-aaea-465f-854e-4d883cad4bd5-kube-api-access-bpzsp\") pod \"authorino-6889cbd8c4-nvhp8\" (UID: \"9ceee7d0-aaea-465f-854e-4d883cad4bd5\") " pod="kuadrant-system/authorino-6889cbd8c4-nvhp8" Oct 30 04:14:15.449814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.449735 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ceee7d0-aaea-465f-854e-4d883cad4bd5-tls-cert\") pod \"authorino-6889cbd8c4-nvhp8\" (UID: \"9ceee7d0-aaea-465f-854e-4d883cad4bd5\") " pod="kuadrant-system/authorino-6889cbd8c4-nvhp8" Oct 30 04:14:15.451736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.451706 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0b2dc7-d28e-4ef4-8dcb-c4467b272089-kube-api-access-2rz45" pod "1b0b2dc7-d28e-4ef4-8dcb-c4467b272089" (UID: "1b0b2dc7-d28e-4ef4-8dcb-c4467b272089"). InnerVolumeSpecName "kube-api-access-2rz45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:14:15.451973 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.451956 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ceee7d0-aaea-465f-854e-4d883cad4bd5-tls-cert\") pod \"authorino-6889cbd8c4-nvhp8\" (UID: \"9ceee7d0-aaea-465f-854e-4d883cad4bd5\") " pod="kuadrant-system/authorino-6889cbd8c4-nvhp8" Oct 30 04:14:15.473302 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.473276 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpzsp\" (UniqueName: \"kubernetes.io/projected/9ceee7d0-aaea-465f-854e-4d883cad4bd5-kube-api-access-bpzsp\") pod \"authorino-6889cbd8c4-nvhp8\" (UID: \"9ceee7d0-aaea-465f-854e-4d883cad4bd5\") " pod="kuadrant-system/authorino-6889cbd8c4-nvhp8" Oct 30 04:14:15.550480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:15.550440 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rz45\" (UniqueName: \"kubernetes.io/projected/1b0b2dc7-d28e-4ef4-8dcb-c4467b272089-kube-api-access-2rz45\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:14:16.232233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.232200 4793 generic.go:356] "Generic (PLEG): container finished" podID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerID="80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8" exitCode=0 Oct 30 04:14:16.232776 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.232273 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" event={"ID":"734b5f28-3d4f-4674-b509-0fb356ee779b","Type":"ContainerDied","Data":"80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8"} Oct 30 04:14:16.232776 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.232323 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-57jln" Oct 30 04:14:16.232776 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.232282 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-nvhp8" Oct 30 04:14:16.232947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.232816 4793 scope.go:122] "RemoveContainer" containerID="80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8" Oct 30 04:14:16.233905 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.233872 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:14:16.234009 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.233940 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:14:16.234009 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.233971 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:14:16.238847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.238828 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-nvhp8" Oct 30 04:14:16.356163 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.356129 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-57jln"] Oct 30 04:14:16.357814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.357790 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-57jln"] Oct 30 04:14:16.357989 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.357971 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ceee7d0-aaea-465f-854e-4d883cad4bd5-tls-cert\" (UniqueName: \"kubernetes.io/secret/9ceee7d0-aaea-465f-854e-4d883cad4bd5-tls-cert\") pod \"9ceee7d0-aaea-465f-854e-4d883cad4bd5\" (UID: \"9ceee7d0-aaea-465f-854e-4d883cad4bd5\") " Oct 30 04:14:16.358082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.358063 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ceee7d0-aaea-465f-854e-4d883cad4bd5-kube-api-access-bpzsp\" (UniqueName: \"kubernetes.io/projected/9ceee7d0-aaea-465f-854e-4d883cad4bd5-kube-api-access-bpzsp\") pod \"9ceee7d0-aaea-465f-854e-4d883cad4bd5\" (UID: \"9ceee7d0-aaea-465f-854e-4d883cad4bd5\") " Oct 30 04:14:16.360473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.360446 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ceee7d0-aaea-465f-854e-4d883cad4bd5-tls-cert" pod "9ceee7d0-aaea-465f-854e-4d883cad4bd5" (UID: "9ceee7d0-aaea-465f-854e-4d883cad4bd5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:14:16.360897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.360871 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ceee7d0-aaea-465f-854e-4d883cad4bd5-kube-api-access-bpzsp" pod "9ceee7d0-aaea-465f-854e-4d883cad4bd5" (UID: "9ceee7d0-aaea-465f-854e-4d883cad4bd5"). InnerVolumeSpecName "kube-api-access-bpzsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:14:16.459828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.459797 4793 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9ceee7d0-aaea-465f-854e-4d883cad4bd5-tls-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:14:16.459828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.459825 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpzsp\" (UniqueName: \"kubernetes.io/projected/9ceee7d0-aaea-465f-854e-4d883cad4bd5-kube-api-access-bpzsp\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:14:16.843212 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:16.843172 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0b2dc7-d28e-4ef4-8dcb-c4467b272089" path="/var/lib/kubelet/pods/1b0b2dc7-d28e-4ef4-8dcb-c4467b272089/volumes" Oct 30 04:14:17.223156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.223099 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m6qxl"] Oct 30 04:14:17.223761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.223706 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" podUID="a0f7828b-c780-4dad-80bb-c42935eafee1" containerName="authorino" containerID="cri-o://dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4" gracePeriod=30 Oct 30 04:14:17.241987 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.241962 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" event={"ID":"734b5f28-3d4f-4674-b509-0fb356ee779b","Type":"ContainerStarted","Data":"380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00"} Oct 30 04:14:17.242435 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.242000 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-nvhp8" Oct 30 04:14:17.388434 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.388377 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nvhp8"] Oct 30 04:14:17.394611 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.394578 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-nvhp8"] Oct 30 04:14:17.531819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.531792 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" Oct 30 04:14:17.585914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.585882 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f7828b-c780-4dad-80bb-c42935eafee1-kube-api-access-vbd57\" (UniqueName: \"kubernetes.io/projected/a0f7828b-c780-4dad-80bb-c42935eafee1-kube-api-access-vbd57\") pod \"a0f7828b-c780-4dad-80bb-c42935eafee1\" (UID: \"a0f7828b-c780-4dad-80bb-c42935eafee1\") " Oct 30 04:14:17.588309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.588280 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f7828b-c780-4dad-80bb-c42935eafee1-kube-api-access-vbd57" pod "a0f7828b-c780-4dad-80bb-c42935eafee1" (UID: "a0f7828b-c780-4dad-80bb-c42935eafee1"). InnerVolumeSpecName "kube-api-access-vbd57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:14:17.686709 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.686678 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbd57\" (UniqueName: \"kubernetes.io/projected/a0f7828b-c780-4dad-80bb-c42935eafee1-kube-api-access-vbd57\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:14:17.720717 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.720678 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:14:17.720855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.720827 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9vqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.15:15021/healthz/ready\": dial tcp 10.133.0.15:15021: connect: connection refused" start-of-body= Oct 30 04:14:17.720911 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:17.720883 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.15:15021/healthz/ready\": dial tcp 10.133.0.15:15021: connect: connection refused" Oct 30 04:14:18.249367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.249337 4793 generic.go:356] "Generic (PLEG): container finished" podID="a0f7828b-c780-4dad-80bb-c42935eafee1" containerID="dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4" exitCode=0 Oct 30 04:14:18.249840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.249381 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" Oct 30 04:14:18.249840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.249432 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" event={"ID":"a0f7828b-c780-4dad-80bb-c42935eafee1","Type":"ContainerDied","Data":"dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4"} Oct 30 04:14:18.249840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.249456 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-m6qxl" event={"ID":"a0f7828b-c780-4dad-80bb-c42935eafee1","Type":"ContainerDied","Data":"1a4f38ffc164620600dcef2d84106740d94553b31e4ceffa72624b1c5c8daff8"} Oct 30 04:14:18.249840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.249474 4793 scope.go:122] "RemoveContainer" containerID="dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4" Oct 30 04:14:18.283901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.283583 4793 scope.go:122] "RemoveContainer" containerID="dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4" Oct 30 04:14:18.284046 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:14:18.284026 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4\": container with ID starting with dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4 not found: ID does not exist" containerID="dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4" Oct 30 04:14:18.284109 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.284059 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4"} err="failed to get container status \"dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4\": rpc error: code = NotFound desc = could not find container \"dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4\": container with ID starting with dc971761b20ecf8e14e3d3dfe1020c4fca2922119ca0184574109a8b2bce96c4 not found: ID does not exist" Oct 30 04:14:18.320563 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.320540 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m6qxl"] Oct 30 04:14:18.324952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.324932 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-m6qxl"] Oct 30 04:14:18.720738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.720700 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9vqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.15:15021/healthz/ready\": dial tcp 10.133.0.15:15021: connect: connection refused" start-of-body= Oct 30 04:14:18.720931 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.720759 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.15:15021/healthz/ready\": dial tcp 10.133.0.15:15021: connect: connection refused" Oct 30 04:14:18.844265 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.844231 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ceee7d0-aaea-465f-854e-4d883cad4bd5" path="/var/lib/kubelet/pods/9ceee7d0-aaea-465f-854e-4d883cad4bd5/volumes" Oct 30 04:14:18.844482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:18.844469 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f7828b-c780-4dad-80bb-c42935eafee1" path="/var/lib/kubelet/pods/a0f7828b-c780-4dad-80bb-c42935eafee1/volumes" Oct 30 04:14:19.720091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:19.720055 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9vqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.15:15021/healthz/ready\": dial tcp 10.133.0.15:15021: connect: connection refused" start-of-body= Oct 30 04:14:19.720531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:19.720110 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.15:15021/healthz/ready\": dial tcp 10.133.0.15:15021: connect: connection refused" Oct 30 04:14:20.720196 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:20.720166 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9vqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.15:15021/healthz/ready\": dial tcp 10.133.0.15:15021: connect: connection refused" start-of-body= Oct 30 04:14:20.720568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:20.720217 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.15:15021/healthz/ready\": dial tcp 10.133.0.15:15021: connect: connection refused" Oct 30 04:14:21.721399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:21.721363 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9vqv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.15:15021/healthz/ready\": dial tcp 10.133.0.15:15021: connect: connection refused" start-of-body= Oct 30 04:14:21.721801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:21.721438 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.15:15021/healthz/ready\": dial tcp 10.133.0.15:15021: connect: connection refused" Oct 30 04:14:22.724037 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:22.724003 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:14:22.724648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:22.724441 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:14:22.725100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:22.725080 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:14:33.498230 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.498193 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl"] Oct 30 04:14:33.498680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.498538 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a0f7828b-c780-4dad-80bb-c42935eafee1" containerName="authorino" Oct 30 04:14:33.498680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.498550 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f7828b-c780-4dad-80bb-c42935eafee1" containerName="authorino" Oct 30 04:14:33.498680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.498632 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f7828b-c780-4dad-80bb-c42935eafee1" containerName="authorino" Oct 30 04:14:33.501290 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.501273 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.535981 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.535951 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl"] Oct 30 04:14:33.592358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.592328 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.592358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.592360 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcxnq\" (UniqueName: \"kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-kube-api-access-bcxnq\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.592547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.592387 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.592547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.592515 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.592618 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.592557 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.592618 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.592577 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.592680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.592641 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.592680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.592668 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.592769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.592693 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.693448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.693407 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.693587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.693457 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.693587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.693479 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.693587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.693495 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcxnq\" (UniqueName: \"kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-kube-api-access-bcxnq\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.693587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.693519 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.693587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.693535 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.693587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.693562 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.693587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.693583 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.693888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.693621 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.693936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.693882 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.693936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.693908 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.694051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.694027 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.694112 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.694063 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.694435 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.694399 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.696052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.696019 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.696160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.696138 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.721454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.721429 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcxnq\" (UniqueName: \"kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-kube-api-access-bcxnq\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.732394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.732367 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78prl\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:33.813752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:33.813704 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:34.131497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:34.131463 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl"] Oct 30 04:14:34.133014 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:14:34.132977 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13cc193_7d5e_4a91_a3e2_e00572f2dbd1.slice/crio-0267324bd02785b5ff32091cad755584c928f91ca65e0f924677d5f4b20c699b WatchSource:0}: Error finding container 0267324bd02785b5ff32091cad755584c928f91ca65e0f924677d5f4b20c699b: Status 404 returned error can't find the container with id 0267324bd02785b5ff32091cad755584c928f91ca65e0f924677d5f4b20c699b Oct 30 04:14:34.135027 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:34.134992 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:14:34.135108 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:34.135058 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:14:34.135108 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:34.135089 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:14:34.346258 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:34.346220 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" event={"ID":"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1","Type":"ContainerStarted","Data":"9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a"} Oct 30 04:14:34.346467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:34.346280 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" event={"ID":"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1","Type":"ContainerStarted","Data":"0267324bd02785b5ff32091cad755584c928f91ca65e0f924677d5f4b20c699b"} Oct 30 04:14:34.394142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:34.394052 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" podStartSLOduration=1.394037369 podStartE2EDuration="1.394037369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:34.391779196 +0000 UTC m=+997.677653976" watchObservedRunningTime="2026-10-30 04:14:34.394037369 +0000 UTC m=+997.679912143" Oct 30 04:14:34.814119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:34.814082 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:35.816053 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:35.816018 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78prl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 04:14:35.816448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:35.816068 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" podUID="a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 04:14:36.814282 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:36.814245 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78prl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 04:14:36.814463 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:36.814295 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" podUID="a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 04:14:37.815082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:37.815048 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78prl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 04:14:37.815521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:37.815100 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" podUID="a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 04:14:38.814211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:38.814173 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78prl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 04:14:38.814441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:38.814233 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" podUID="a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 04:14:39.814087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:39.814050 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78prl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 04:14:39.814532 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:39.814099 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" podUID="a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 04:14:40.817559 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:40.817525 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:40.817985 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:40.817788 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:40.818353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:40.818336 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 04:14:41.218163 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:14:41.218126 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:14:48.545490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.545460 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57"] Oct 30 04:14:48.547706 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.547687 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.570724 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.570694 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57"] Oct 30 04:14:48.620696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.620665 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.620813 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.620704 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.620813 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.620730 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.620888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.620823 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.620922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.620905 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.620955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.620927 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rbdf\" (UniqueName: \"kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-kube-api-access-7rbdf\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.621033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.620993 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.621067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.621050 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.621100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.621072 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.721783 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.721755 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.721898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.721792 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.721898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.721824 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.721898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.721851 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.721898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.721886 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.722080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.721909 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rbdf\" (UniqueName: \"kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-kube-api-access-7rbdf\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.722080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.722014 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.722222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.722196 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.722292 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.722256 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.722349 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.722323 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.722406 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.722348 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.722500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.722414 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.722587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.722566 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.722834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.722818 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.724247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.724225 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.724335 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.724278 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.744975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.744956 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.745236 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.745214 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rbdf\" (UniqueName: \"kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-kube-api-access-7rbdf\") pod \"maas-default-gateway-openshift-default-7f694df8bc-scw57\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:48.861358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:48.861303 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:49.170752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:49.170720 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57"] Oct 30 04:14:49.172079 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:14:49.172052 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b0ea95_77b3_46d5_9a4e_dca7b91efa76.slice/crio-e3e6f20d9b023efd4f76a16649991700571de633474c8057901df0cfe7e32aa5 WatchSource:0}: Error finding container e3e6f20d9b023efd4f76a16649991700571de633474c8057901df0cfe7e32aa5: Status 404 returned error can't find the container with id e3e6f20d9b023efd4f76a16649991700571de633474c8057901df0cfe7e32aa5 Oct 30 04:14:49.174192 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:49.174159 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:14:49.174291 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:49.174222 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:14:49.174291 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:49.174254 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:14:49.442097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:49.442061 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" event={"ID":"21b0ea95-77b3-46d5-9a4e-dca7b91efa76","Type":"ContainerStarted","Data":"ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8"} Oct 30 04:14:49.442282 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:49.442142 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" event={"ID":"21b0ea95-77b3-46d5-9a4e-dca7b91efa76","Type":"ContainerStarted","Data":"e3e6f20d9b023efd4f76a16649991700571de633474c8057901df0cfe7e32aa5"} Oct 30 04:14:49.490751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:49.490699 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" podStartSLOduration=1.490685173 podStartE2EDuration="1.490685173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:49.488540778 +0000 UTC m=+1012.774415554" watchObservedRunningTime="2026-10-30 04:14:49.490685173 +0000 UTC m=+1012.776559947" Oct 30 04:14:49.868636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:49.868526 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:50.868014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:50.867981 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-scw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 04:14:50.868281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:50.868252 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" podUID="21b0ea95-77b3-46d5-9a4e-dca7b91efa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 04:14:51.864824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:51.864785 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-scw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 04:14:51.865440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:51.865390 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" podUID="21b0ea95-77b3-46d5-9a4e-dca7b91efa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 04:14:52.274747 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.274669 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-vtxpc"] Oct 30 04:14:52.283861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.283830 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" Oct 30 04:14:52.284938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.284907 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-vtxpc"] Oct 30 04:14:52.299378 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.299355 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-f57wp\"" Oct 30 04:14:52.299378 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.299370 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 04:14:52.299542 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.299375 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 04:14:52.351774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.351743 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7342d626-179d-400d-9f0e-75e52116fb24-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-vtxpc\" (UID: \"7342d626-179d-400d-9f0e-75e52116fb24\") " pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" Oct 30 04:14:52.351926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.351800 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2grp\" (UniqueName: \"kubernetes.io/projected/7342d626-179d-400d-9f0e-75e52116fb24-kube-api-access-t2grp\") pod \"ai-gateway-controller-7d85675f98-vtxpc\" (UID: \"7342d626-179d-400d-9f0e-75e52116fb24\") " pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" Oct 30 04:14:52.452291 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.452252 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7342d626-179d-400d-9f0e-75e52116fb24-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-vtxpc\" (UID: \"7342d626-179d-400d-9f0e-75e52116fb24\") " pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" Oct 30 04:14:52.452479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.452311 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2grp\" (UniqueName: \"kubernetes.io/projected/7342d626-179d-400d-9f0e-75e52116fb24-kube-api-access-t2grp\") pod \"ai-gateway-controller-7d85675f98-vtxpc\" (UID: \"7342d626-179d-400d-9f0e-75e52116fb24\") " pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" Oct 30 04:14:52.479319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.479285 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7342d626-179d-400d-9f0e-75e52116fb24-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-vtxpc\" (UID: \"7342d626-179d-400d-9f0e-75e52116fb24\") " pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" Oct 30 04:14:52.481845 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.481818 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2grp\" (UniqueName: \"kubernetes.io/projected/7342d626-179d-400d-9f0e-75e52116fb24-kube-api-access-t2grp\") pod \"ai-gateway-controller-7d85675f98-vtxpc\" (UID: \"7342d626-179d-400d-9f0e-75e52116fb24\") " pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" Oct 30 04:14:52.596026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.595953 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" Oct 30 04:14:52.859147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.859065 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:52.878878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.878842 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-scw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 04:14:52.879313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.878938 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" podUID="21b0ea95-77b3-46d5-9a4e-dca7b91efa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 04:14:52.899414 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:52.899391 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-vtxpc"] Oct 30 04:14:52.900837 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:14:52.900813 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7342d626_179d_400d_9f0e_75e52116fb24.slice/crio-6b143e5964a4988541475d1d5ae1b3cdb9f62bd413cbf2fcc068d6d7693a2f6e WatchSource:0}: Error finding container 6b143e5964a4988541475d1d5ae1b3cdb9f62bd413cbf2fcc068d6d7693a2f6e: Status 404 returned error can't find the container with id 6b143e5964a4988541475d1d5ae1b3cdb9f62bd413cbf2fcc068d6d7693a2f6e Oct 30 04:14:53.125029 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:53.124947 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:53.483747 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:53.483699 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" event={"ID":"7342d626-179d-400d-9f0e-75e52116fb24","Type":"ContainerStarted","Data":"6b143e5964a4988541475d1d5ae1b3cdb9f62bd413cbf2fcc068d6d7693a2f6e"} Oct 30 04:14:53.880646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:53.880559 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-scw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 04:14:53.881134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:53.880648 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" podUID="21b0ea95-77b3-46d5-9a4e-dca7b91efa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 04:14:54.874231 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:54.873787 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-scw57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Oct 30 04:14:54.874231 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:54.873846 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" podUID="21b0ea95-77b3-46d5-9a4e-dca7b91efa76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Oct 30 04:14:55.865176 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:55.865146 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:55.865558 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:55.865496 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:55.866134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:55.866117 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:14:56.501330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:56.501297 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" event={"ID":"7342d626-179d-400d-9f0e-75e52116fb24","Type":"ContainerStarted","Data":"39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1"} Oct 30 04:14:56.501538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:56.501524 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" Oct 30 04:14:56.544919 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:56.544863 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" podStartSLOduration=1.8070672650000001 podStartE2EDuration="4.5448462s" totalImagesPullingTime="2.737778935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:14:56.542181292 +0000 UTC m=+1019.828056067" watchObservedRunningTime="2026-10-30 04:14:56.5448462 +0000 UTC m=+1019.830720978" Oct 30 04:14:57.857122 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:57.857085 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-l4r5r"] Oct 30 04:14:57.860169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:57.860148 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" Oct 30 04:14:57.878321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:57.878297 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cn9lf\"" Oct 30 04:14:57.880242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:57.880223 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 04:14:57.880552 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:57.880530 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-l4r5r"] Oct 30 04:14:57.915200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:57.915171 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:57.923180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:57.923155 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv"] Oct 30 04:14:57.925499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:57.925482 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" Oct 30 04:14:57.926559 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:57.926540 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db99ac60-acfe-44f7-9729-9ef0ef451830-plugins-config-volume\") pod \"payload-processing-5f78687d5b-l4r5r\" (UID: \"db99ac60-acfe-44f7-9729-9ef0ef451830\") " pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" Oct 30 04:14:57.926686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:57.926668 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79ddd\" (UniqueName: \"kubernetes.io/projected/db99ac60-acfe-44f7-9729-9ef0ef451830-kube-api-access-79ddd\") pod \"payload-processing-5f78687d5b-l4r5r\" (UID: \"db99ac60-acfe-44f7-9729-9ef0ef451830\") " pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" Oct 30 04:14:57.947393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:57.947367 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv"] Oct 30 04:14:58.028103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.028053 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79ddd\" (UniqueName: \"kubernetes.io/projected/db99ac60-acfe-44f7-9729-9ef0ef451830-kube-api-access-79ddd\") pod \"payload-processing-5f78687d5b-l4r5r\" (UID: \"db99ac60-acfe-44f7-9729-9ef0ef451830\") " pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" Oct 30 04:14:58.028103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.028100 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vn6g\" (UniqueName: \"kubernetes.io/projected/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-kube-api-access-6vn6g\") pod \"payload-pre-processing-7b98bb678c-v7lnv\" (UID: \"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" Oct 30 04:14:58.028433 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.028227 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db99ac60-acfe-44f7-9729-9ef0ef451830-plugins-config-volume\") pod \"payload-processing-5f78687d5b-l4r5r\" (UID: \"db99ac60-acfe-44f7-9729-9ef0ef451830\") " pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" Oct 30 04:14:58.028433 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.028344 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-v7lnv\" (UID: \"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" Oct 30 04:14:58.029028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.029001 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db99ac60-acfe-44f7-9729-9ef0ef451830-plugins-config-volume\") pod \"payload-processing-5f78687d5b-l4r5r\" (UID: \"db99ac60-acfe-44f7-9729-9ef0ef451830\") " pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" Oct 30 04:14:58.052095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.052052 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79ddd\" (UniqueName: \"kubernetes.io/projected/db99ac60-acfe-44f7-9729-9ef0ef451830-kube-api-access-79ddd\") pod \"payload-processing-5f78687d5b-l4r5r\" (UID: \"db99ac60-acfe-44f7-9729-9ef0ef451830\") " pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" Oct 30 04:14:58.129500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.128901 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vn6g\" (UniqueName: \"kubernetes.io/projected/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-kube-api-access-6vn6g\") pod \"payload-pre-processing-7b98bb678c-v7lnv\" (UID: \"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" Oct 30 04:14:58.129500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.129015 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-v7lnv\" (UID: \"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" Oct 30 04:14:58.133260 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.130040 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-v7lnv\" (UID: \"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" Oct 30 04:14:58.154472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.154431 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:14:58.166768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.166740 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vn6g\" (UniqueName: \"kubernetes.io/projected/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-kube-api-access-6vn6g\") pod \"payload-pre-processing-7b98bb678c-v7lnv\" (UID: \"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" Oct 30 04:14:58.176698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.176669 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" Oct 30 04:14:58.236684 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.236647 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" Oct 30 04:14:58.501402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.501380 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-l4r5r"] Oct 30 04:14:58.502255 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:14:58.502228 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb99ac60_acfe_44f7_9729_9ef0ef451830.slice/crio-f59a6742ef50c0c49302dfeb0a74c1deba267363660a4ca183ca57b3e5a1d122 WatchSource:0}: Error finding container f59a6742ef50c0c49302dfeb0a74c1deba267363660a4ca183ca57b3e5a1d122: Status 404 returned error can't find the container with id f59a6742ef50c0c49302dfeb0a74c1deba267363660a4ca183ca57b3e5a1d122 Oct 30 04:14:58.512031 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.512007 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" event={"ID":"db99ac60-acfe-44f7-9729-9ef0ef451830","Type":"ContainerStarted","Data":"f59a6742ef50c0c49302dfeb0a74c1deba267363660a4ca183ca57b3e5a1d122"} Oct 30 04:14:58.548882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:58.548860 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv"] Oct 30 04:14:58.551030 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:14:58.551006 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09ef3d4_79e1_4a47_8b15_0b863ef31ff3.slice/crio-1d2b0f4ebe11a3d931e909e4654c72e7c9e5df81dcd9f299a586672030b062b0 WatchSource:0}: Error finding container 1d2b0f4ebe11a3d931e909e4654c72e7c9e5df81dcd9f299a586672030b062b0: Status 404 returned error can't find the container with id 1d2b0f4ebe11a3d931e909e4654c72e7c9e5df81dcd9f299a586672030b062b0 Oct 30 04:14:59.519383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:14:59.519343 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" event={"ID":"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3","Type":"ContainerStarted","Data":"1d2b0f4ebe11a3d931e909e4654c72e7c9e5df81dcd9f299a586672030b062b0"} Oct 30 04:15:00.153851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.153808 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt"] Oct 30 04:15:00.163954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.163499 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" Oct 30 04:15:00.175736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.175710 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt"] Oct 30 04:15:00.185903 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.185876 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:15:00.186038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.185927 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:15:00.186038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.185953 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:15:00.248132 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.248096 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31c23c6-1e59-40b2-82d8-44919c465631-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888895-56xbt\" (UID: \"a31c23c6-1e59-40b2-82d8-44919c465631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" Oct 30 04:15:00.248328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.248160 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxq9g\" (UniqueName: \"kubernetes.io/projected/a31c23c6-1e59-40b2-82d8-44919c465631-kube-api-access-hxq9g\") pod \"maas-api-key-cleanup-29888895-56xbt\" (UID: \"a31c23c6-1e59-40b2-82d8-44919c465631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" Oct 30 04:15:00.351935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.351897 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31c23c6-1e59-40b2-82d8-44919c465631-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888895-56xbt\" (UID: \"a31c23c6-1e59-40b2-82d8-44919c465631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" Oct 30 04:15:00.352162 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.352135 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxq9g\" (UniqueName: \"kubernetes.io/projected/a31c23c6-1e59-40b2-82d8-44919c465631-kube-api-access-hxq9g\") pod \"maas-api-key-cleanup-29888895-56xbt\" (UID: \"a31c23c6-1e59-40b2-82d8-44919c465631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" Oct 30 04:15:00.352914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.352885 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31c23c6-1e59-40b2-82d8-44919c465631-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888895-56xbt\" (UID: \"a31c23c6-1e59-40b2-82d8-44919c465631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" Oct 30 04:15:00.377061 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.377005 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxq9g\" (UniqueName: \"kubernetes.io/projected/a31c23c6-1e59-40b2-82d8-44919c465631-kube-api-access-hxq9g\") pod \"maas-api-key-cleanup-29888895-56xbt\" (UID: \"a31c23c6-1e59-40b2-82d8-44919c465631\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" Oct 30 04:15:00.482121 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.482089 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" Oct 30 04:15:00.803526 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:15:00.803499 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda31c23c6_1e59_40b2_82d8_44919c465631.slice/crio-05b8f9043e8bf33476bf268261e893585211bd1b92ac7d13c7898d5e1cc1113c WatchSource:0}: Error finding container 05b8f9043e8bf33476bf268261e893585211bd1b92ac7d13c7898d5e1cc1113c: Status 404 returned error can't find the container with id 05b8f9043e8bf33476bf268261e893585211bd1b92ac7d13c7898d5e1cc1113c Oct 30 04:15:00.812734 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:00.812709 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt"] Oct 30 04:15:01.531260 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:01.531219 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" event={"ID":"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3","Type":"ContainerStarted","Data":"a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c"} Oct 30 04:15:01.531452 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:01.531361 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" Oct 30 04:15:01.532561 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:01.532530 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" event={"ID":"a31c23c6-1e59-40b2-82d8-44919c465631","Type":"ContainerStarted","Data":"05b8f9043e8bf33476bf268261e893585211bd1b92ac7d13c7898d5e1cc1113c"} Oct 30 04:15:01.533867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:01.533846 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" event={"ID":"db99ac60-acfe-44f7-9729-9ef0ef451830","Type":"ContainerStarted","Data":"0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0"} Oct 30 04:15:01.533973 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:01.533885 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" Oct 30 04:15:01.572538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:01.572498 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" podStartSLOduration=2.37360899 podStartE2EDuration="4.572486787s" totalImagesPullingTime="2.198877797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:01.572087687 +0000 UTC m=+1024.857962483" watchObservedRunningTime="2026-10-30 04:15:01.572486787 +0000 UTC m=+1024.858361557" Oct 30 04:15:01.613391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:01.613355 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" podStartSLOduration=2.453786465 podStartE2EDuration="4.613344857s" totalImagesPullingTime="2.159558392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:01.612619085 +0000 UTC m=+1024.898493859" watchObservedRunningTime="2026-10-30 04:15:01.613344857 +0000 UTC m=+1024.899219631" Oct 30 04:15:03.547887 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.547851 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" event={"ID":"a31c23c6-1e59-40b2-82d8-44919c465631","Type":"ContainerStarted","Data":"4445895db541069ed8f6277ae8cfcced6fd29bbb3e455a41c9a4b1c16305dfb9"} Oct 30 04:15:03.585091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.585061 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g"] Oct 30 04:15:03.587501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.587476 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.598126 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.598083 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" podStartSLOduration=1.441911858 podStartE2EDuration="3.598070648s" totalImagesPullingTime="2.15615879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:03.596643167 +0000 UTC m=+1026.882517941" watchObservedRunningTime="2026-10-30 04:15:03.598070648 +0000 UTC m=+1026.883945422" Oct 30 04:15:03.622925 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.622898 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g"] Oct 30 04:15:03.678934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.678906 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.679033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.678961 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.679033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.679023 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.679104 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.679057 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.679181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.679157 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e816986-f54e-44d3-b431-0d09094fa6ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.679240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.679228 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.679274 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.679258 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.679307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.679289 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24nns\" (UniqueName: \"kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-kube-api-access-24nns\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.679340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.679326 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.779957 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.779929 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.779962 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.779981 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.780011 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.780064 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e816986-f54e-44d3-b431-0d09094fa6ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.780215 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.780260 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780408 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.780304 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nns\" (UniqueName: \"kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-kube-api-access-24nns\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780408 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.780335 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780408 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.780349 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780588 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.780472 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780588 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.780494 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.780612 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.780929 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.780912 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e816986-f54e-44d3-b431-0d09094fa6ff-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.782344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.782324 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.782648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.782631 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.808681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.808633 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.811051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.811028 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nns\" (UniqueName: \"kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-kube-api-access-24nns\") pod \"maas-default-gateway-openshift-default-7f694df8bc-w6h4g\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:03.900016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:03.899994 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:04.210862 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:04.210840 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g"] Oct 30 04:15:04.212060 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:15:04.212026 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e816986_f54e_44d3_b431_0d09094fa6ff.slice/crio-5c9287bb4bef2b3667930133dda387ad57b9e69b45b38b6fc7f72a8bb01d0efe WatchSource:0}: Error finding container 5c9287bb4bef2b3667930133dda387ad57b9e69b45b38b6fc7f72a8bb01d0efe: Status 404 returned error can't find the container with id 5c9287bb4bef2b3667930133dda387ad57b9e69b45b38b6fc7f72a8bb01d0efe Oct 30 04:15:04.214318 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:04.214284 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:15:04.214473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:04.214359 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:15:04.214473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:04.214403 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:15:04.559817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:04.559699 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" event={"ID":"2e816986-f54e-44d3-b431-0d09094fa6ff","Type":"ContainerStarted","Data":"e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54"} Oct 30 04:15:04.559817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:04.559803 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" event={"ID":"2e816986-f54e-44d3-b431-0d09094fa6ff","Type":"ContainerStarted","Data":"5c9287bb4bef2b3667930133dda387ad57b9e69b45b38b6fc7f72a8bb01d0efe"} Oct 30 04:15:04.562169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:04.562136 4793 generic.go:356] "Generic (PLEG): container finished" podID="a31c23c6-1e59-40b2-82d8-44919c465631" containerID="4445895db541069ed8f6277ae8cfcced6fd29bbb3e455a41c9a4b1c16305dfb9" exitCode=60 Oct 30 04:15:04.562324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:04.562240 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" event={"ID":"a31c23c6-1e59-40b2-82d8-44919c465631","Type":"ContainerDied","Data":"4445895db541069ed8f6277ae8cfcced6fd29bbb3e455a41c9a4b1c16305dfb9"} Oct 30 04:15:04.562640 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:04.562623 4793 scope.go:122] "RemoveContainer" containerID="4445895db541069ed8f6277ae8cfcced6fd29bbb3e455a41c9a4b1c16305dfb9" Oct 30 04:15:04.619592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:04.619548 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" podStartSLOduration=1.6195338750000001 podStartE2EDuration="1.619533875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:04.617929629 +0000 UTC m=+1027.903804405" watchObservedRunningTime="2026-10-30 04:15:04.619533875 +0000 UTC m=+1027.905408649" Oct 30 04:15:04.900436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:04.900390 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:05.584905 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:05.584869 4793 generic.go:356] "Generic (PLEG): container finished" podID="a31c23c6-1e59-40b2-82d8-44919c465631" containerID="ec4a61361ff1adcd7aa7a89545410ca52552552462e4bbf430a1506fff270481" exitCode=60 Oct 30 04:15:05.585268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:05.584950 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" event={"ID":"a31c23c6-1e59-40b2-82d8-44919c465631","Type":"ContainerDied","Data":"ec4a61361ff1adcd7aa7a89545410ca52552552462e4bbf430a1506fff270481"} Oct 30 04:15:05.585268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:05.585012 4793 scope.go:122] "RemoveContainer" containerID="4445895db541069ed8f6277ae8cfcced6fd29bbb3e455a41c9a4b1c16305dfb9" Oct 30 04:15:05.585268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:05.585203 4793 scope.go:122] "RemoveContainer" containerID="ec4a61361ff1adcd7aa7a89545410ca52552552462e4bbf430a1506fff270481" Oct 30 04:15:05.585476 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:15:05.585453 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888895-56xbt_odh-ai-gateway-infra(a31c23c6-1e59-40b2-82d8-44919c465631)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" Oct 30 04:15:05.902230 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:05.902146 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-w6h4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 04:15:05.902230 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:05.902200 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" podUID="2e816986-f54e-44d3-b431-0d09094fa6ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 04:15:06.591983 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:06.591951 4793 scope.go:122] "RemoveContainer" containerID="ec4a61361ff1adcd7aa7a89545410ca52552552462e4bbf430a1506fff270481" Oct 30 04:15:06.592469 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:15:06.592215 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888895-56xbt_odh-ai-gateway-infra(a31c23c6-1e59-40b2-82d8-44919c465631)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" Oct 30 04:15:06.901119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:06.901024 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-w6h4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 04:15:06.901119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:06.901088 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" podUID="2e816986-f54e-44d3-b431-0d09094fa6ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 04:15:07.508784 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:07.508748 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" Oct 30 04:15:07.813292 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:07.813220 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl"] Oct 30 04:15:07.816251 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:07.816229 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" Oct 30 04:15:07.836280 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:07.836260 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl"] Oct 30 04:15:07.901053 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:07.901021 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-w6h4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 04:15:07.901214 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:07.901069 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" podUID="2e816986-f54e-44d3-b431-0d09094fa6ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 04:15:07.917443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:07.917399 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfx5f\" (UniqueName: \"kubernetes.io/projected/a736cf03-a95d-4e46-9358-7b8e6bba483b-kube-api-access-rfx5f\") pod \"ai-gateway-controller-79d77bcfdb-6hwnl\" (UID: \"a736cf03-a95d-4e46-9358-7b8e6bba483b\") " pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" Oct 30 04:15:07.917615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:07.917597 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736cf03-a95d-4e46-9358-7b8e6bba483b-metrics-certs\") pod \"ai-gateway-controller-79d77bcfdb-6hwnl\" (UID: \"a736cf03-a95d-4e46-9358-7b8e6bba483b\") " pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" Oct 30 04:15:08.019090 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:08.019059 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736cf03-a95d-4e46-9358-7b8e6bba483b-metrics-certs\") pod \"ai-gateway-controller-79d77bcfdb-6hwnl\" (UID: \"a736cf03-a95d-4e46-9358-7b8e6bba483b\") " pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" Oct 30 04:15:08.019233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:08.019118 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfx5f\" (UniqueName: \"kubernetes.io/projected/a736cf03-a95d-4e46-9358-7b8e6bba483b-kube-api-access-rfx5f\") pod \"ai-gateway-controller-79d77bcfdb-6hwnl\" (UID: \"a736cf03-a95d-4e46-9358-7b8e6bba483b\") " pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" Oct 30 04:15:08.021460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:08.021436 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a736cf03-a95d-4e46-9358-7b8e6bba483b-metrics-certs\") pod \"ai-gateway-controller-79d77bcfdb-6hwnl\" (UID: \"a736cf03-a95d-4e46-9358-7b8e6bba483b\") " pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" Oct 30 04:15:08.042671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:08.042650 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfx5f\" (UniqueName: \"kubernetes.io/projected/a736cf03-a95d-4e46-9358-7b8e6bba483b-kube-api-access-rfx5f\") pod \"ai-gateway-controller-79d77bcfdb-6hwnl\" (UID: \"a736cf03-a95d-4e46-9358-7b8e6bba483b\") " pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" Oct 30 04:15:08.127496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:08.127255 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" Oct 30 04:15:08.445015 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:08.444988 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl"] Oct 30 04:15:08.445531 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:15:08.445501 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda736cf03_a95d_4e46_9358_7b8e6bba483b.slice/crio-0ace345187c52d6043428bc4bb4ca03a25e9847d05df4b68ae71d5c87f7cceeb WatchSource:0}: Error finding container 0ace345187c52d6043428bc4bb4ca03a25e9847d05df4b68ae71d5c87f7cceeb: Status 404 returned error can't find the container with id 0ace345187c52d6043428bc4bb4ca03a25e9847d05df4b68ae71d5c87f7cceeb Oct 30 04:15:08.602918 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:08.602885 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" event={"ID":"a736cf03-a95d-4e46-9358-7b8e6bba483b","Type":"ContainerStarted","Data":"0ace345187c52d6043428bc4bb4ca03a25e9847d05df4b68ae71d5c87f7cceeb"} Oct 30 04:15:08.900794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:08.900771 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-w6h4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 04:15:08.901097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:08.900817 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" podUID="2e816986-f54e-44d3-b431-0d09094fa6ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 04:15:09.609288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:09.609199 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" event={"ID":"a736cf03-a95d-4e46-9358-7b8e6bba483b","Type":"ContainerStarted","Data":"88e1aed6a9004876076f33111167580a378f0856c06786a457fda1f2a74d8229"} Oct 30 04:15:09.609571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:09.609340 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" Oct 30 04:15:09.652216 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:09.652169 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" podStartSLOduration=2.278864715 podStartE2EDuration="2.652156444s" totalImagesPullingTime="373.291729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:09.651366172 +0000 UTC m=+1032.937240946" watchObservedRunningTime="2026-10-30 04:15:09.652156444 +0000 UTC m=+1032.938031218" Oct 30 04:15:09.900394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:09.900317 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-w6h4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 04:15:09.900394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:09.900366 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" podUID="2e816986-f54e-44d3-b431-0d09094fa6ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 04:15:10.904322 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:10.904295 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:10.904732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:10.904552 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:10.905293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:10.905274 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:15:11.218230 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:15:11.218196 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:15:12.544096 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:12.544066 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" Oct 30 04:15:12.544470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:12.544127 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" Oct 30 04:15:14.396847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:14.396806 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:15.838275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:15.838244 4793 scope.go:122] "RemoveContainer" containerID="ec4a61361ff1adcd7aa7a89545410ca52552552462e4bbf430a1506fff270481" Oct 30 04:15:16.648112 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:16.648077 4793 generic.go:356] "Generic (PLEG): container finished" podID="a31c23c6-1e59-40b2-82d8-44919c465631" containerID="b1141c460c88c929a6aa2a2caf2efeead42c885bbc1bfd5fe1fb947613c6c4f1" exitCode=60 Oct 30 04:15:16.648267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:16.648125 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" event={"ID":"a31c23c6-1e59-40b2-82d8-44919c465631","Type":"ContainerDied","Data":"b1141c460c88c929a6aa2a2caf2efeead42c885bbc1bfd5fe1fb947613c6c4f1"} Oct 30 04:15:16.648267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:16.648172 4793 scope.go:122] "RemoveContainer" containerID="ec4a61361ff1adcd7aa7a89545410ca52552552462e4bbf430a1506fff270481" Oct 30 04:15:16.648543 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:16.648523 4793 scope.go:122] "RemoveContainer" containerID="b1141c460c88c929a6aa2a2caf2efeead42c885bbc1bfd5fe1fb947613c6c4f1" Oct 30 04:15:16.648793 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:15:16.648766 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888895-56xbt_odh-ai-gateway-infra(a31c23c6-1e59-40b2-82d8-44919c465631)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" Oct 30 04:15:17.700653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:17.700622 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt"] Oct 30 04:15:17.833338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:17.833316 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" Oct 30 04:15:18.012055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:18.011977 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a31c23c6-1e59-40b2-82d8-44919c465631-kube-api-access-hxq9g\" (UniqueName: \"kubernetes.io/projected/a31c23c6-1e59-40b2-82d8-44919c465631-kube-api-access-hxq9g\") pod \"a31c23c6-1e59-40b2-82d8-44919c465631\" (UID: \"a31c23c6-1e59-40b2-82d8-44919c465631\") " Oct 30 04:15:18.012055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:18.012017 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a31c23c6-1e59-40b2-82d8-44919c465631-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31c23c6-1e59-40b2-82d8-44919c465631-maas-api-service-ca\") pod \"a31c23c6-1e59-40b2-82d8-44919c465631\" (UID: \"a31c23c6-1e59-40b2-82d8-44919c465631\") " Oct 30 04:15:18.012394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:18.012363 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a31c23c6-1e59-40b2-82d8-44919c465631-maas-api-service-ca" pod "a31c23c6-1e59-40b2-82d8-44919c465631" (UID: "a31c23c6-1e59-40b2-82d8-44919c465631"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:15:18.014098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:18.014081 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a31c23c6-1e59-40b2-82d8-44919c465631-kube-api-access-hxq9g" pod "a31c23c6-1e59-40b2-82d8-44919c465631" (UID: "a31c23c6-1e59-40b2-82d8-44919c465631"). InnerVolumeSpecName "kube-api-access-hxq9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:15:18.112653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:18.112628 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxq9g\" (UniqueName: \"kubernetes.io/projected/a31c23c6-1e59-40b2-82d8-44919c465631-kube-api-access-hxq9g\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:15:18.112653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:18.112649 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a31c23c6-1e59-40b2-82d8-44919c465631-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:15:18.660798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:18.660772 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" Oct 30 04:15:18.660960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:18.660766 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt" event={"ID":"a31c23c6-1e59-40b2-82d8-44919c465631","Type":"ContainerDied","Data":"05b8f9043e8bf33476bf268261e893585211bd1b92ac7d13c7898d5e1cc1113c"} Oct 30 04:15:18.660960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:18.660908 4793 scope.go:122] "RemoveContainer" containerID="b1141c460c88c929a6aa2a2caf2efeead42c885bbc1bfd5fe1fb947613c6c4f1" Oct 30 04:15:18.712775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:18.712752 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt"] Oct 30 04:15:18.725353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:18.725330 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888895-56xbt"] Oct 30 04:15:18.843540 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:18.843511 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" path="/var/lib/kubelet/pods/a31c23c6-1e59-40b2-82d8-44919c465631/volumes" Oct 30 04:15:20.616536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:20.616508 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-79d77bcfdb-6hwnl" Oct 30 04:15:20.687165 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:20.687140 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-vtxpc"] Oct 30 04:15:20.687371 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:20.687336 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" podUID="7342d626-179d-400d-9f0e-75e52116fb24" containerName="manager" containerID="cri-o://39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1" gracePeriod=30 Oct 30 04:15:20.931755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:20.931733 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" Oct 30 04:15:21.038949 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.038920 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7342d626-179d-400d-9f0e-75e52116fb24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7342d626-179d-400d-9f0e-75e52116fb24-metrics-certs\") pod \"7342d626-179d-400d-9f0e-75e52116fb24\" (UID: \"7342d626-179d-400d-9f0e-75e52116fb24\") " Oct 30 04:15:21.039065 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.038992 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7342d626-179d-400d-9f0e-75e52116fb24-kube-api-access-t2grp\" (UniqueName: \"kubernetes.io/projected/7342d626-179d-400d-9f0e-75e52116fb24-kube-api-access-t2grp\") pod \"7342d626-179d-400d-9f0e-75e52116fb24\" (UID: \"7342d626-179d-400d-9f0e-75e52116fb24\") " Oct 30 04:15:21.041030 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.041003 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7342d626-179d-400d-9f0e-75e52116fb24-metrics-certs" pod "7342d626-179d-400d-9f0e-75e52116fb24" (UID: "7342d626-179d-400d-9f0e-75e52116fb24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:15:21.041138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.041120 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7342d626-179d-400d-9f0e-75e52116fb24-kube-api-access-t2grp" pod "7342d626-179d-400d-9f0e-75e52116fb24" (UID: "7342d626-179d-400d-9f0e-75e52116fb24"). InnerVolumeSpecName "kube-api-access-t2grp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:15:21.140369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.140340 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7342d626-179d-400d-9f0e-75e52116fb24-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:15:21.140369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.140365 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2grp\" (UniqueName: \"kubernetes.io/projected/7342d626-179d-400d-9f0e-75e52116fb24-kube-api-access-t2grp\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:15:21.678608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.678574 4793 generic.go:356] "Generic (PLEG): container finished" podID="7342d626-179d-400d-9f0e-75e52116fb24" containerID="39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1" exitCode=0 Oct 30 04:15:21.678608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.678609 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" event={"ID":"7342d626-179d-400d-9f0e-75e52116fb24","Type":"ContainerDied","Data":"39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1"} Oct 30 04:15:21.679033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.678620 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" Oct 30 04:15:21.679033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.678648 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-vtxpc" event={"ID":"7342d626-179d-400d-9f0e-75e52116fb24","Type":"ContainerDied","Data":"6b143e5964a4988541475d1d5ae1b3cdb9f62bd413cbf2fcc068d6d7693a2f6e"} Oct 30 04:15:21.679033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.678670 4793 scope.go:122] "RemoveContainer" containerID="39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1" Oct 30 04:15:21.687634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.687612 4793 scope.go:122] "RemoveContainer" containerID="39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1" Oct 30 04:15:21.687874 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:15:21.687858 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1\": container with ID starting with 39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1 not found: ID does not exist" containerID="39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1" Oct 30 04:15:21.687927 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.687881 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1"} err="failed to get container status \"39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1\": rpc error: code = NotFound desc = could not find container \"39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1\": container with ID starting with 39fe6953d46dad4503ba2539e0435416ce62ed45560eb51b58657a124f3e60d1 not found: ID does not exist" Oct 30 04:15:21.731451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.731430 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-vtxpc"] Oct 30 04:15:21.743933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:21.743913 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-vtxpc"] Oct 30 04:15:22.614711 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:22.614675 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:22.843245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:22.843212 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7342d626-179d-400d-9f0e-75e52116fb24" path="/var/lib/kubelet/pods/7342d626-179d-400d-9f0e-75e52116fb24/volumes" Oct 30 04:15:23.209146 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:23.209115 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:24.517054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:24.517018 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:24.844663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:24.844569 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:29.148960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.148931 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk"] Oct 30 04:15:29.149313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.149269 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" containerName="cleanup" Oct 30 04:15:29.149313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.149281 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" containerName="cleanup" Oct 30 04:15:29.149313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.149299 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" containerName="cleanup" Oct 30 04:15:29.149313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.149304 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" containerName="cleanup" Oct 30 04:15:29.149313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.149314 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" containerName="cleanup" Oct 30 04:15:29.149502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.149321 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" containerName="cleanup" Oct 30 04:15:29.149502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.149332 4793 cpu_manager.go:414] "containerMap: removing container" podUID="7342d626-179d-400d-9f0e-75e52116fb24" containerName="manager" Oct 30 04:15:29.149502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.149338 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7342d626-179d-400d-9f0e-75e52116fb24" containerName="manager" Oct 30 04:15:29.149502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.149389 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" containerName="cleanup" Oct 30 04:15:29.149502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.149398 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" containerName="cleanup" Oct 30 04:15:29.149502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.149404 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a31c23c6-1e59-40b2-82d8-44919c465631" containerName="cleanup" Oct 30 04:15:29.149502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.149411 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="7342d626-179d-400d-9f0e-75e52116fb24" containerName="manager" Oct 30 04:15:29.156860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.156838 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.171721 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.171695 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk"] Oct 30 04:15:29.182534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.182510 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dx86c\"" Oct 30 04:15:29.182666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.182597 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 04:15:29.182666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.182633 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 04:15:29.184694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.184675 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 04:15:29.203487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.203461 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.203593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.203498 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.203593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.203523 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.203668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.203638 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb2dn\" (UniqueName: \"kubernetes.io/projected/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-kube-api-access-bb2dn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.203702 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.203691 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.203735 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.203713 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.304278 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.304248 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.304278 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.304280 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.304493 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.304300 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.304493 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.304326 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb2dn\" (UniqueName: \"kubernetes.io/projected/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-kube-api-access-bb2dn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.304493 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.304389 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.304493 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.304437 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.304681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.304659 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.304793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.304773 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.304856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.304836 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.306617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.306595 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.306731 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.306714 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.310262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.310185 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:29.329434 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.329397 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb2dn\" (UniqueName: \"kubernetes.io/projected/ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b-kube-api-access-bb2dn\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk\" (UID: \"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.469603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.469574 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:29.776679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:29.776647 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk"] Oct 30 04:15:29.777545 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:15:29.777517 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce835b60_df8c_4ee0_a2a8_e3bb44b37b2b.slice/crio-a2fa3242775aca72e5ba026066be4ad54d3a148d60fc330a208a2088dc9dc264 WatchSource:0}: Error finding container a2fa3242775aca72e5ba026066be4ad54d3a148d60fc330a208a2088dc9dc264: Status 404 returned error can't find the container with id a2fa3242775aca72e5ba026066be4ad54d3a148d60fc330a208a2088dc9dc264 Oct 30 04:15:30.730136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:30.730014 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" event={"ID":"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b","Type":"ContainerStarted","Data":"a2fa3242775aca72e5ba026066be4ad54d3a148d60fc330a208a2088dc9dc264"} Oct 30 04:15:31.737330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:31.737292 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" event={"ID":"ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b","Type":"ContainerStarted","Data":"c1c9f517dfc2bdfca8bc0e545d27a091726867c606bfa3da42a446e180935fe9"} Oct 30 04:15:31.781560 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:31.781511 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" podStartSLOduration=1.242782928 podStartE2EDuration="2.781497628s" totalImagesPullingTime="1.5387147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:31.77969705 +0000 UTC m=+1055.065571836" watchObservedRunningTime="2026-10-30 04:15:31.781497628 +0000 UTC m=+1055.067372442" Oct 30 04:15:32.341548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.341516 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx"] Oct 30 04:15:32.345327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.345210 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.366266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.366234 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx"] Oct 30 04:15:32.373337 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.373311 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 04:15:32.436473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.436446 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fb589e-1545-4aba-a706-ee3532bbfe65-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.436607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.436494 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fb589e-1545-4aba-a706-ee3532bbfe65-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.436607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.436517 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxzgk\" (UniqueName: \"kubernetes.io/projected/05fb589e-1545-4aba-a706-ee3532bbfe65-kube-api-access-wxzgk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.436706 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.436669 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fb589e-1545-4aba-a706-ee3532bbfe65-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.436764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.436704 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fb589e-1545-4aba-a706-ee3532bbfe65-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.436764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.436734 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fb589e-1545-4aba-a706-ee3532bbfe65-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.469882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.469858 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:32.503281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.503258 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:32.537993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.537963 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fb589e-1545-4aba-a706-ee3532bbfe65-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.538194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.538031 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fb589e-1545-4aba-a706-ee3532bbfe65-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.538194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.538100 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fb589e-1545-4aba-a706-ee3532bbfe65-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.538194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.538126 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxzgk\" (UniqueName: \"kubernetes.io/projected/05fb589e-1545-4aba-a706-ee3532bbfe65-kube-api-access-wxzgk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.538468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.538225 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fb589e-1545-4aba-a706-ee3532bbfe65-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.538468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.538248 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fb589e-1545-4aba-a706-ee3532bbfe65-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.539614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.538790 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fb589e-1545-4aba-a706-ee3532bbfe65-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.539614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.539319 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fb589e-1545-4aba-a706-ee3532bbfe65-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.539614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.539581 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fb589e-1545-4aba-a706-ee3532bbfe65-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.542114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.542004 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fb589e-1545-4aba-a706-ee3532bbfe65-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.542502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.542398 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fb589e-1545-4aba-a706-ee3532bbfe65-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.562913 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.562883 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxzgk\" (UniqueName: \"kubernetes.io/projected/05fb589e-1545-4aba-a706-ee3532bbfe65-kube-api-access-wxzgk\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-xxxsx\" (UID: \"05fb589e-1545-4aba-a706-ee3532bbfe65\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.660398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.660323 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:32.963836 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:32.963807 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx"] Oct 30 04:15:32.964665 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:15:32.964634 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fb589e_1545_4aba_a706_ee3532bbfe65.slice/crio-c2d5938c7b92f68ceae0a5be12fbbf0a1b38aa04365a6fd72fc791f8b86b8362 WatchSource:0}: Error finding container c2d5938c7b92f68ceae0a5be12fbbf0a1b38aa04365a6fd72fc791f8b86b8362: Status 404 returned error can't find the container with id c2d5938c7b92f68ceae0a5be12fbbf0a1b38aa04365a6fd72fc791f8b86b8362 Oct 30 04:15:33.776471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:33.767864 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" event={"ID":"05fb589e-1545-4aba-a706-ee3532bbfe65","Type":"ContainerStarted","Data":"14fa19d5ae7419ac4729455f7ed7359533213a21905d25f6399d6fcc5c66fb34"} Oct 30 04:15:33.776471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:33.767922 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" event={"ID":"05fb589e-1545-4aba-a706-ee3532bbfe65","Type":"ContainerStarted","Data":"c2d5938c7b92f68ceae0a5be12fbbf0a1b38aa04365a6fd72fc791f8b86b8362"} Oct 30 04:15:33.830441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:33.826023 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" podStartSLOduration=1.6437786330000002 podStartE2EDuration="1.826005481s" totalImagesPullingTime="182.226848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:33.82584257 +0000 UTC m=+1057.111717346" watchObservedRunningTime="2026-10-30 04:15:33.826005481 +0000 UTC m=+1057.111880256" Oct 30 04:15:34.660602 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:34.660570 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:35.504008 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:35.503977 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:37.997072 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:37.997040 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt"] Oct 30 04:15:38.006860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.004126 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.021567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.020552 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt"] Oct 30 04:15:38.036698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.034659 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac0e332-6063-4c8c-9445-0f3bd4946caf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.036698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.034812 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac0e332-6063-4c8c-9445-0f3bd4946caf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.036698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.034895 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tgpm\" (UniqueName: \"kubernetes.io/projected/bac0e332-6063-4c8c-9445-0f3bd4946caf-kube-api-access-7tgpm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.036698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.035050 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac0e332-6063-4c8c-9445-0f3bd4946caf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.036698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.035189 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac0e332-6063-4c8c-9445-0f3bd4946caf-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.036698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.035226 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac0e332-6063-4c8c-9445-0f3bd4946caf-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.036698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.035397 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 04:15:38.136309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.136274 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac0e332-6063-4c8c-9445-0f3bd4946caf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.136475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.136336 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac0e332-6063-4c8c-9445-0f3bd4946caf-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.136475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.136450 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac0e332-6063-4c8c-9445-0f3bd4946caf-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.136569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.136488 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac0e332-6063-4c8c-9445-0f3bd4946caf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.136569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.136530 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac0e332-6063-4c8c-9445-0f3bd4946caf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.136671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.136566 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tgpm\" (UniqueName: \"kubernetes.io/projected/bac0e332-6063-4c8c-9445-0f3bd4946caf-kube-api-access-7tgpm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.136778 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.136757 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bac0e332-6063-4c8c-9445-0f3bd4946caf-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.136895 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.136875 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bac0e332-6063-4c8c-9445-0f3bd4946caf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.136994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.136966 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bac0e332-6063-4c8c-9445-0f3bd4946caf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.138712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.138687 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bac0e332-6063-4c8c-9445-0f3bd4946caf-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.139002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.138981 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bac0e332-6063-4c8c-9445-0f3bd4946caf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.160447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.160408 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tgpm\" (UniqueName: \"kubernetes.io/projected/bac0e332-6063-4c8c-9445-0f3bd4946caf-kube-api-access-7tgpm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wpknt\" (UID: \"bac0e332-6063-4c8c-9445-0f3bd4946caf\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.164378 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.164348 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:38.342998 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.342917 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:38.638540 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.638518 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt"] Oct 30 04:15:38.639990 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:15:38.639963 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac0e332_6063_4c8c_9445_0f3bd4946caf.slice/crio-0a5c7889ac0da4441515f539553a10ed6fb757a2066f9cf2988e97c669a32f00 WatchSource:0}: Error finding container 0a5c7889ac0da4441515f539553a10ed6fb757a2066f9cf2988e97c669a32f00: Status 404 returned error can't find the container with id 0a5c7889ac0da4441515f539553a10ed6fb757a2066f9cf2988e97c669a32f00 Oct 30 04:15:38.799683 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:38.799651 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" event={"ID":"bac0e332-6063-4c8c-9445-0f3bd4946caf","Type":"ContainerStarted","Data":"0a5c7889ac0da4441515f539553a10ed6fb757a2066f9cf2988e97c669a32f00"} Oct 30 04:15:39.473286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:39.472497 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:39.490675 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:39.490645 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:39.809350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:39.809209 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" event={"ID":"bac0e332-6063-4c8c-9445-0f3bd4946caf","Type":"ContainerStarted","Data":"1fabc8eaeb78dcfc10bc28a43d2e44d8e53cfd0e0546ac50e1d42b5fa4bcbbdb"} Oct 30 04:15:39.828769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:39.828742 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk" Oct 30 04:15:39.858871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:39.857008 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" podStartSLOduration=2.691952942 podStartE2EDuration="2.856989321s" totalImagesPullingTime="165.036379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:39.854875225 +0000 UTC m=+1063.140749996" watchObservedRunningTime="2026-10-30 04:15:39.856989321 +0000 UTC m=+1063.142864099" Oct 30 04:15:39.884899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:39.884865 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:40.352547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:40.349514 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:41.217903 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:15:41.217866 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:15:42.463101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:42.463056 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:42.661495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:42.661456 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:42.681844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:42.681816 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:42.851994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:42.851901 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-xxxsx" Oct 30 04:15:44.908233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:44.908205 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr"] Oct 30 04:15:44.912064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:44.912039 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:44.941990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:44.941955 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 04:15:44.944093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:44.944071 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr"] Oct 30 04:15:44.957600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:44.957569 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4bb1e95-2059-43dd-8abf-384abf8cdefb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:44.957844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:44.957814 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bb1e95-2059-43dd-8abf-384abf8cdefb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:44.957958 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:44.957861 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgph5\" (UniqueName: \"kubernetes.io/projected/b4bb1e95-2059-43dd-8abf-384abf8cdefb-kube-api-access-pgph5\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:44.957958 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:44.957906 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4bb1e95-2059-43dd-8abf-384abf8cdefb-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:44.957958 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:44.957940 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4bb1e95-2059-43dd-8abf-384abf8cdefb-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:44.958096 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:44.957985 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4bb1e95-2059-43dd-8abf-384abf8cdefb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.050899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.050858 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:45.059063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.059038 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4bb1e95-2059-43dd-8abf-384abf8cdefb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.059154 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.059074 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bb1e95-2059-43dd-8abf-384abf8cdefb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.059154 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.059095 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgph5\" (UniqueName: \"kubernetes.io/projected/b4bb1e95-2059-43dd-8abf-384abf8cdefb-kube-api-access-pgph5\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.059154 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.059126 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4bb1e95-2059-43dd-8abf-384abf8cdefb-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.059275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.059160 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4bb1e95-2059-43dd-8abf-384abf8cdefb-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.059275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.059202 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4bb1e95-2059-43dd-8abf-384abf8cdefb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.059600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.059575 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4bb1e95-2059-43dd-8abf-384abf8cdefb-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.059651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.059592 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4bb1e95-2059-43dd-8abf-384abf8cdefb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.059651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.059590 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4bb1e95-2059-43dd-8abf-384abf8cdefb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.061559 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.061538 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4bb1e95-2059-43dd-8abf-384abf8cdefb-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.061629 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.061617 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4bb1e95-2059-43dd-8abf-384abf8cdefb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.082542 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.082517 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgph5\" (UniqueName: \"kubernetes.io/projected/b4bb1e95-2059-43dd-8abf-384abf8cdefb-kube-api-access-pgph5\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr\" (UID: \"b4bb1e95-2059-43dd-8abf-384abf8cdefb\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.226116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.226064 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:45.579074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.579044 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr"] Oct 30 04:15:45.582067 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:15:45.582036 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bb1e95_2059_43dd_8abf_384abf8cdefb.slice/crio-a8b4a07544924235ff771ea4e6075dd5794c11ed3b015c90654ce261bd59135a WatchSource:0}: Error finding container a8b4a07544924235ff771ea4e6075dd5794c11ed3b015c90654ce261bd59135a: Status 404 returned error can't find the container with id a8b4a07544924235ff771ea4e6075dd5794c11ed3b015c90654ce261bd59135a Oct 30 04:15:45.909849 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.909717 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" event={"ID":"b4bb1e95-2059-43dd-8abf-384abf8cdefb","Type":"ContainerStarted","Data":"bf90e12c7dfb715e0640a4e224d0649175dc854df900175ee1e42bdca8915b67"} Oct 30 04:15:45.909849 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:45.909811 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" event={"ID":"b4bb1e95-2059-43dd-8abf-384abf8cdefb","Type":"ContainerStarted","Data":"a8b4a07544924235ff771ea4e6075dd5794c11ed3b015c90654ce261bd59135a"} Oct 30 04:15:46.232441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:46.226496 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:48.343491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.343453 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:48.355988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.355965 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:48.403387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.403332 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" podStartSLOduration=4.219624646 podStartE2EDuration="4.403319028s" totalImagesPullingTime="183.694382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:45.973481314 +0000 UTC m=+1069.259356090" watchObservedRunningTime="2026-10-30 04:15:48.403319028 +0000 UTC m=+1071.689193804" Oct 30 04:15:48.418541 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.418502 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp"] Oct 30 04:15:48.422502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.422481 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.433810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.433782 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp"] Oct 30 04:15:48.444405 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.444384 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 04:15:48.491690 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.491666 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5097d128-40d6-4c8e-9b0b-6269985b6bed-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.491789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.491728 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqw4l\" (UniqueName: \"kubernetes.io/projected/5097d128-40d6-4c8e-9b0b-6269985b6bed-kube-api-access-lqw4l\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.491840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.491783 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5097d128-40d6-4c8e-9b0b-6269985b6bed-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.491876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.491851 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5097d128-40d6-4c8e-9b0b-6269985b6bed-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.491914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.491880 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5097d128-40d6-4c8e-9b0b-6269985b6bed-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.491963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.491949 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5097d128-40d6-4c8e-9b0b-6269985b6bed-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.562142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.562112 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:48.593863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.593124 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5097d128-40d6-4c8e-9b0b-6269985b6bed-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.593863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.593230 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqw4l\" (UniqueName: \"kubernetes.io/projected/5097d128-40d6-4c8e-9b0b-6269985b6bed-kube-api-access-lqw4l\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.593863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.593271 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5097d128-40d6-4c8e-9b0b-6269985b6bed-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.593863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.593312 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5097d128-40d6-4c8e-9b0b-6269985b6bed-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.593863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.593351 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5097d128-40d6-4c8e-9b0b-6269985b6bed-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.593863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.593392 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5097d128-40d6-4c8e-9b0b-6269985b6bed-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.595289 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.594331 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5097d128-40d6-4c8e-9b0b-6269985b6bed-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.595289 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.594643 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5097d128-40d6-4c8e-9b0b-6269985b6bed-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.595289 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.595228 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5097d128-40d6-4c8e-9b0b-6269985b6bed-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.602180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.602154 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5097d128-40d6-4c8e-9b0b-6269985b6bed-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.602565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.602544 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5097d128-40d6-4c8e-9b0b-6269985b6bed-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.617908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.617883 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqw4l\" (UniqueName: \"kubernetes.io/projected/5097d128-40d6-4c8e-9b0b-6269985b6bed-kube-api-access-lqw4l\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp\" (UID: \"5097d128-40d6-4c8e-9b0b-6269985b6bed\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.742960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.742918 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:48.964276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:48.964241 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wpknt" Oct 30 04:15:49.067364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:49.067305 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp"] Oct 30 04:15:50.004442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:50.003797 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" event={"ID":"5097d128-40d6-4c8e-9b0b-6269985b6bed","Type":"ContainerStarted","Data":"cd2e04fbbd8707ad06d9750b3225562f994499512c22988ea8274d829b3cb842"} Oct 30 04:15:50.004442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:50.003847 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" event={"ID":"5097d128-40d6-4c8e-9b0b-6269985b6bed","Type":"ContainerStarted","Data":"a0999a32ef1e6cb7a3cf7c3c15236406d20a7436d05084d17186dd8c793dd86b"} Oct 30 04:15:50.057441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:50.057270 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" podStartSLOduration=1.882251361 podStartE2EDuration="2.057250008s" totalImagesPullingTime="174.998647ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:15:50.050778601 +0000 UTC m=+1073.336653377" watchObservedRunningTime="2026-10-30 04:15:50.057250008 +0000 UTC m=+1073.343124784" Oct 30 04:15:50.746074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:50.744499 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:51.944320 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:51.944280 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:15:53.773444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:53.772545 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9vqv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:15:53.773444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:53.772609 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:15:55.226988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:55.226956 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:55.239848 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:55.239822 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:56.151572 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:56.151545 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr" Oct 30 04:15:58.743906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:58.743873 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:58.756348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:58.756325 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:15:59.166250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:15:59.166175 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp" Oct 30 04:16:08.172139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:08.172102 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:08.379807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:08.379774 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:09.758208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:09.758173 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:09.980866 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:09.980828 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:11.218848 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:16:11.218810 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:16:41.217829 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:16:41.217744 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:16:41.561703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:41.561627 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:41.824113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:41.824032 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:42.119019 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:42.118932 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:42.335313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:42.335275 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:42.545021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:42.544992 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:42.818391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:42.818291 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:43.049271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.049237 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:43.470681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.470645 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:43.781628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.781536 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl"] Oct 30 04:16:43.785614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.785593 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:43.808772 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.808743 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl"] Oct 30 04:16:43.846671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.846644 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ad07bf73-openshift-default-dockercfg-ntmzn\"" Oct 30 04:16:43.907180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.907148 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-envoy\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:43.907304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.907201 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ab6828-4959-4412-9fdb-50237628cd8d-istiod-ca-cert\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:43.907304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.907273 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-data\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:43.907396 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.907324 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-podinfo\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:43.907445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.907409 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-certs\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:43.907495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.907454 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rkq4\" (UniqueName: \"kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-kube-api-access-7rkq4\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:43.907544 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.907519 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-socket\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:43.907579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.907554 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-credential-socket\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:43.907623 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.907592 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-token\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:43.967824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:43.967794 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:44.008801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.008759 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ab6828-4959-4412-9fdb-50237628cd8d-istiod-ca-cert\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.009899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.009870 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-data\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.010117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.010099 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-podinfo\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.010273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.010256 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-certs\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.010400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.010383 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rkq4\" (UniqueName: \"kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-kube-api-access-7rkq4\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.010613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.010592 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-socket\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.010797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.010767 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-credential-socket\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.010903 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.010328 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-data\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.011331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.010934 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-token\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.011480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.011391 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-envoy\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.011556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.009790 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ab6828-4959-4412-9fdb-50237628cd8d-istiod-ca-cert\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.011556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.010609 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-certs\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.011556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.011164 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-credential-socket\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.011556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.010952 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-socket\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.014220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.014198 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-envoy\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.019916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.019895 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-podinfo\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.034125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.034067 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-token\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.034125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.034092 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rkq4\" (UniqueName: \"kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-kube-api-access-7rkq4\") pod \"e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.101725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.101704 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:44.182071 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.182035 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:44.442245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.442214 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl"] Oct 30 04:16:44.443389 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:16:44.443359 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ab6828_4959_4412_9fdb_50237628cd8d.slice/crio-91eaf2dfabcf355eca7c238f11e0d73e8e16613b7215baa51bec5da451730ac6 WatchSource:0}: Error finding container 91eaf2dfabcf355eca7c238f11e0d73e8e16613b7215baa51bec5da451730ac6: Status 404 returned error can't find the container with id 91eaf2dfabcf355eca7c238f11e0d73e8e16613b7215baa51bec5da451730ac6 Oct 30 04:16:44.445477 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.445444 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:16:44.445583 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.445509 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:16:44.445583 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.445546 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:16:44.720046 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.719961 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:44.928120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:44.928085 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:45.148722 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:45.148639 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:45.407247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:45.407159 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" event={"ID":"d8ab6828-4959-4412-9fdb-50237628cd8d","Type":"ContainerStarted","Data":"4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5"} Oct 30 04:16:45.407247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:45.407220 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" event={"ID":"d8ab6828-4959-4412-9fdb-50237628cd8d","Type":"ContainerStarted","Data":"91eaf2dfabcf355eca7c238f11e0d73e8e16613b7215baa51bec5da451730ac6"} Oct 30 04:16:45.480676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:45.480619 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" podStartSLOduration=2.480606424 podStartE2EDuration="2.480606424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:45.466929193 +0000 UTC m=+1128.752804002" watchObservedRunningTime="2026-10-30 04:16:45.480606424 +0000 UTC m=+1128.766481199" Oct 30 04:16:46.101916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.101882 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:46.106831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.106807 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:46.376150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.376048 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:46.412712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.412686 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:46.413519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.413499 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:16:46.431585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.431552 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz"] Oct 30 04:16:46.435033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.435016 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" Oct 30 04:16:46.453270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.453251 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-eb861bd9\"" Oct 30 04:16:46.453463 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.453345 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-eb861bd9-dockercfg-qh8pn\"" Oct 30 04:16:46.454689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.454670 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz"] Oct 30 04:16:46.510235 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.510201 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh"] Oct 30 04:16:46.513676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.513652 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" Oct 30 04:16:46.535158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.535128 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh"] Oct 30 04:16:46.535675 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.535648 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0863945a-4d72-4b2d-9c18-da05eb39bd48-plugins-config-volume\") pod \"payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz\" (UID: \"0863945a-4d72-4b2d-9c18-da05eb39bd48\") " pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" Oct 30 04:16:46.535907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.535885 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrdtv\" (UniqueName: \"kubernetes.io/projected/0863945a-4d72-4b2d-9c18-da05eb39bd48-kube-api-access-rrdtv\") pod \"payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz\" (UID: \"0863945a-4d72-4b2d-9c18-da05eb39bd48\") " pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" Oct 30 04:16:46.576689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.576631 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:46.642105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.642014 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9t7\" (UniqueName: \"kubernetes.io/projected/32651cda-6e75-4992-bfd2-92d9db399459-kube-api-access-cq9t7\") pod \"payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh\" (UID: \"32651cda-6e75-4992-bfd2-92d9db399459\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" Oct 30 04:16:46.642394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.642361 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0863945a-4d72-4b2d-9c18-da05eb39bd48-plugins-config-volume\") pod \"payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz\" (UID: \"0863945a-4d72-4b2d-9c18-da05eb39bd48\") " pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" Oct 30 04:16:46.642496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.642445 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdtv\" (UniqueName: \"kubernetes.io/projected/0863945a-4d72-4b2d-9c18-da05eb39bd48-kube-api-access-rrdtv\") pod \"payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz\" (UID: \"0863945a-4d72-4b2d-9c18-da05eb39bd48\") " pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" Oct 30 04:16:46.642496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.642477 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32651cda-6e75-4992-bfd2-92d9db399459-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh\" (UID: \"32651cda-6e75-4992-bfd2-92d9db399459\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" Oct 30 04:16:46.643358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.643331 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0863945a-4d72-4b2d-9c18-da05eb39bd48-plugins-config-volume\") pod \"payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz\" (UID: \"0863945a-4d72-4b2d-9c18-da05eb39bd48\") " pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" Oct 30 04:16:46.666763 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.666735 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrdtv\" (UniqueName: \"kubernetes.io/projected/0863945a-4d72-4b2d-9c18-da05eb39bd48-kube-api-access-rrdtv\") pod \"payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz\" (UID: \"0863945a-4d72-4b2d-9c18-da05eb39bd48\") " pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" Oct 30 04:16:46.743670 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.743637 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32651cda-6e75-4992-bfd2-92d9db399459-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh\" (UID: \"32651cda-6e75-4992-bfd2-92d9db399459\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" Oct 30 04:16:46.743801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.743736 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9t7\" (UniqueName: \"kubernetes.io/projected/32651cda-6e75-4992-bfd2-92d9db399459-kube-api-access-cq9t7\") pod \"payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh\" (UID: \"32651cda-6e75-4992-bfd2-92d9db399459\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" Oct 30 04:16:46.744352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.744331 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32651cda-6e75-4992-bfd2-92d9db399459-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh\" (UID: \"32651cda-6e75-4992-bfd2-92d9db399459\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" Oct 30 04:16:46.748498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.748479 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" Oct 30 04:16:46.767982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.767961 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9t7\" (UniqueName: \"kubernetes.io/projected/32651cda-6e75-4992-bfd2-92d9db399459-kube-api-access-cq9t7\") pod \"payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh\" (UID: \"32651cda-6e75-4992-bfd2-92d9db399459\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" Oct 30 04:16:46.828803 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:46.828769 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" Oct 30 04:16:47.059150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:47.059120 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz"] Oct 30 04:16:47.059973 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:16:47.059946 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0863945a_4d72_4b2d_9c18_da05eb39bd48.slice/crio-586f989d6c18eeb76bb1bc8f699367e3bf9d7dbaf31beb68d77af29805353089 WatchSource:0}: Error finding container 586f989d6c18eeb76bb1bc8f699367e3bf9d7dbaf31beb68d77af29805353089: Status 404 returned error can't find the container with id 586f989d6c18eeb76bb1bc8f699367e3bf9d7dbaf31beb68d77af29805353089 Oct 30 04:16:47.143305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:47.143275 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh"] Oct 30 04:16:47.146594 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:16:47.146563 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32651cda_6e75_4992_bfd2_92d9db399459.slice/crio-b21222ee60a30a50f2ac6848b918a3403bfb2378c93c859d81840d8d9ce91f59 WatchSource:0}: Error finding container b21222ee60a30a50f2ac6848b918a3403bfb2378c93c859d81840d8d9ce91f59: Status 404 returned error can't find the container with id b21222ee60a30a50f2ac6848b918a3403bfb2378c93c859d81840d8d9ce91f59 Oct 30 04:16:47.421593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:47.421492 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" event={"ID":"32651cda-6e75-4992-bfd2-92d9db399459","Type":"ContainerStarted","Data":"9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73"} Oct 30 04:16:47.421593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:47.421563 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" event={"ID":"32651cda-6e75-4992-bfd2-92d9db399459","Type":"ContainerStarted","Data":"b21222ee60a30a50f2ac6848b918a3403bfb2378c93c859d81840d8d9ce91f59"} Oct 30 04:16:47.421857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:47.421687 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" Oct 30 04:16:47.423272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:47.423248 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" event={"ID":"0863945a-4d72-4b2d-9c18-da05eb39bd48","Type":"ContainerStarted","Data":"963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1"} Oct 30 04:16:47.423407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:47.423292 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" event={"ID":"0863945a-4d72-4b2d-9c18-da05eb39bd48","Type":"ContainerStarted","Data":"586f989d6c18eeb76bb1bc8f699367e3bf9d7dbaf31beb68d77af29805353089"} Oct 30 04:16:47.464673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:47.464614 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" podStartSLOduration=1.464593246 podStartE2EDuration="1.464593246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:47.462000264 +0000 UTC m=+1130.747875070" watchObservedRunningTime="2026-10-30 04:16:47.464593246 +0000 UTC m=+1130.750468023" Oct 30 04:16:47.506669 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:47.506621 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" podStartSLOduration=1.5066077839999998 podStartE2EDuration="1.506607784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:47.505133372 +0000 UTC m=+1130.791008147" watchObservedRunningTime="2026-10-30 04:16:47.506607784 +0000 UTC m=+1130.792482558" Oct 30 04:16:47.538172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:47.538139 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:47.763861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:47.763819 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:48.179297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.179238 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:48.298331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.298300 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx"] Oct 30 04:16:48.302050 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.302029 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" Oct 30 04:16:48.309138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.309112 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt"] Oct 30 04:16:48.312597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.312579 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" Oct 30 04:16:48.323443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.323404 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-de5733f9\"" Oct 30 04:16:48.323568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.323539 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-de5733f9-dockercfg-gvsbx\"" Oct 30 04:16:48.328465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.328027 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx"] Oct 30 04:16:48.341312 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.341288 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt"] Oct 30 04:16:48.361535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.361500 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6rr\" (UniqueName: \"kubernetes.io/projected/ab80769a-d645-4f51-bc10-f22ea546f798-kube-api-access-5z6rr\") pod \"payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx\" (UID: \"ab80769a-d645-4f51-bc10-f22ea546f798\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" Oct 30 04:16:48.361728 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.361584 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab80769a-d645-4f51-bc10-f22ea546f798-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx\" (UID: \"ab80769a-d645-4f51-bc10-f22ea546f798\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" Oct 30 04:16:48.413771 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.413731 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:48.428633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.428601 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" Oct 30 04:16:48.462886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.462810 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab80769a-d645-4f51-bc10-f22ea546f798-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx\" (UID: \"ab80769a-d645-4f51-bc10-f22ea546f798\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" Oct 30 04:16:48.463025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.462923 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8a8f22-0902-4338-8991-7eaeeb963120-plugins-config-volume\") pod \"payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt\" (UID: \"9b8a8f22-0902-4338-8991-7eaeeb963120\") " pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" Oct 30 04:16:48.463025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.463011 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6rr\" (UniqueName: \"kubernetes.io/projected/ab80769a-d645-4f51-bc10-f22ea546f798-kube-api-access-5z6rr\") pod \"payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx\" (UID: \"ab80769a-d645-4f51-bc10-f22ea546f798\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" Oct 30 04:16:48.463106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.463038 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhgs7\" (UniqueName: \"kubernetes.io/projected/9b8a8f22-0902-4338-8991-7eaeeb963120-kube-api-access-jhgs7\") pod \"payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt\" (UID: \"9b8a8f22-0902-4338-8991-7eaeeb963120\") " pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" Oct 30 04:16:48.463793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.463769 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab80769a-d645-4f51-bc10-f22ea546f798-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx\" (UID: \"ab80769a-d645-4f51-bc10-f22ea546f798\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" Oct 30 04:16:48.486971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.486939 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6rr\" (UniqueName: \"kubernetes.io/projected/ab80769a-d645-4f51-bc10-f22ea546f798-kube-api-access-5z6rr\") pod \"payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx\" (UID: \"ab80769a-d645-4f51-bc10-f22ea546f798\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" Oct 30 04:16:48.563804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.563765 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8a8f22-0902-4338-8991-7eaeeb963120-plugins-config-volume\") pod \"payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt\" (UID: \"9b8a8f22-0902-4338-8991-7eaeeb963120\") " pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" Oct 30 04:16:48.563977 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.563906 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhgs7\" (UniqueName: \"kubernetes.io/projected/9b8a8f22-0902-4338-8991-7eaeeb963120-kube-api-access-jhgs7\") pod \"payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt\" (UID: \"9b8a8f22-0902-4338-8991-7eaeeb963120\") " pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" Oct 30 04:16:48.564644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.564617 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8a8f22-0902-4338-8991-7eaeeb963120-plugins-config-volume\") pod \"payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt\" (UID: \"9b8a8f22-0902-4338-8991-7eaeeb963120\") " pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" Oct 30 04:16:48.589620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.589589 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhgs7\" (UniqueName: \"kubernetes.io/projected/9b8a8f22-0902-4338-8991-7eaeeb963120-kube-api-access-jhgs7\") pod \"payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt\" (UID: \"9b8a8f22-0902-4338-8991-7eaeeb963120\") " pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" Oct 30 04:16:48.616545 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.616517 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" Oct 30 04:16:48.636109 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.636081 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" Oct 30 04:16:48.723728 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.723635 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:48.924190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.924160 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx"] Oct 30 04:16:48.925277 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:16:48.925248 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab80769a_d645_4f51_bc10_f22ea546f798.slice/crio-f3ad11a2aef58c81cadde2306aef17305ca52517aa1c71e76c6edcffbe1890ac WatchSource:0}: Error finding container f3ad11a2aef58c81cadde2306aef17305ca52517aa1c71e76c6edcffbe1890ac: Status 404 returned error can't find the container with id f3ad11a2aef58c81cadde2306aef17305ca52517aa1c71e76c6edcffbe1890ac Oct 30 04:16:48.954461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.954433 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:48.975880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:48.975495 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt"] Oct 30 04:16:48.976998 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:16:48.976968 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8a8f22_0902_4338_8991_7eaeeb963120.slice/crio-e83632a1646c30c5fdc659c00dfa0c5616a4b904637563213efe8cbf67635449 WatchSource:0}: Error finding container e83632a1646c30c5fdc659c00dfa0c5616a4b904637563213efe8cbf67635449: Status 404 returned error can't find the container with id e83632a1646c30c5fdc659c00dfa0c5616a4b904637563213efe8cbf67635449 Oct 30 04:16:49.261136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.261054 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:49.270603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.270572 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f"] Oct 30 04:16:49.274296 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.274276 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" Oct 30 04:16:49.295172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.295150 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-566c3724\"" Oct 30 04:16:49.296556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.296537 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f"] Oct 30 04:16:49.373153 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.373116 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqrpm\" (UniqueName: \"kubernetes.io/projected/428462cc-ef9d-411b-a9fa-6681dc2f6540-kube-api-access-fqrpm\") pod \"payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f\" (UID: \"428462cc-ef9d-411b-a9fa-6681dc2f6540\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" Oct 30 04:16:49.373321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.373168 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428462cc-ef9d-411b-a9fa-6681dc2f6540-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f\" (UID: \"428462cc-ef9d-411b-a9fa-6681dc2f6540\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" Oct 30 04:16:49.419028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.418994 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89"] Oct 30 04:16:49.422836 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.422815 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" Oct 30 04:16:49.435960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.435927 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" event={"ID":"ab80769a-d645-4f51-bc10-f22ea546f798","Type":"ContainerStarted","Data":"c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60"} Oct 30 04:16:49.436104 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.435983 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" event={"ID":"ab80769a-d645-4f51-bc10-f22ea546f798","Type":"ContainerStarted","Data":"f3ad11a2aef58c81cadde2306aef17305ca52517aa1c71e76c6edcffbe1890ac"} Oct 30 04:16:49.436104 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.436037 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" Oct 30 04:16:49.437753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.437727 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" event={"ID":"9b8a8f22-0902-4338-8991-7eaeeb963120","Type":"ContainerStarted","Data":"51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb"} Oct 30 04:16:49.437836 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.437795 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" Oct 30 04:16:49.437836 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.437817 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" event={"ID":"9b8a8f22-0902-4338-8991-7eaeeb963120","Type":"ContainerStarted","Data":"e83632a1646c30c5fdc659c00dfa0c5616a4b904637563213efe8cbf67635449"} Oct 30 04:16:49.456074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.456040 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-566c3724-dockercfg-qvltn\"" Oct 30 04:16:49.458585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.458558 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89"] Oct 30 04:16:49.508473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.497496 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428462cc-ef9d-411b-a9fa-6681dc2f6540-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f\" (UID: \"428462cc-ef9d-411b-a9fa-6681dc2f6540\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" Oct 30 04:16:49.508473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.497547 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89h5\" (UniqueName: \"kubernetes.io/projected/c6b4fca1-447f-438a-a27d-2454c6ac54ad-kube-api-access-n89h5\") pod \"payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89\" (UID: \"c6b4fca1-447f-438a-a27d-2454c6ac54ad\") " pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" Oct 30 04:16:49.508473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.497837 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqrpm\" (UniqueName: \"kubernetes.io/projected/428462cc-ef9d-411b-a9fa-6681dc2f6540-kube-api-access-fqrpm\") pod \"payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f\" (UID: \"428462cc-ef9d-411b-a9fa-6681dc2f6540\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" Oct 30 04:16:49.508473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.497870 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4fca1-447f-438a-a27d-2454c6ac54ad-plugins-config-volume\") pod \"payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89\" (UID: \"c6b4fca1-447f-438a-a27d-2454c6ac54ad\") " pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" Oct 30 04:16:49.508473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.498929 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428462cc-ef9d-411b-a9fa-6681dc2f6540-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f\" (UID: \"428462cc-ef9d-411b-a9fa-6681dc2f6540\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" Oct 30 04:16:49.508473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.504639 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:49.526140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.526065 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqrpm\" (UniqueName: \"kubernetes.io/projected/428462cc-ef9d-411b-a9fa-6681dc2f6540-kube-api-access-fqrpm\") pod \"payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f\" (UID: \"428462cc-ef9d-411b-a9fa-6681dc2f6540\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" Oct 30 04:16:49.535565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.535513 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" podStartSLOduration=1.535493626 podStartE2EDuration="1.535493626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:49.532572251 +0000 UTC m=+1132.818447023" watchObservedRunningTime="2026-10-30 04:16:49.535493626 +0000 UTC m=+1132.821368393" Oct 30 04:16:49.579287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.579231 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" podStartSLOduration=1.5792150440000001 podStartE2EDuration="1.579215044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:49.575734001 +0000 UTC m=+1132.861608777" watchObservedRunningTime="2026-10-30 04:16:49.579215044 +0000 UTC m=+1132.865089820" Oct 30 04:16:49.587720 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.587691 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" Oct 30 04:16:49.599134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.599100 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4fca1-447f-438a-a27d-2454c6ac54ad-plugins-config-volume\") pod \"payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89\" (UID: \"c6b4fca1-447f-438a-a27d-2454c6ac54ad\") " pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" Oct 30 04:16:49.599267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.599156 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89h5\" (UniqueName: \"kubernetes.io/projected/c6b4fca1-447f-438a-a27d-2454c6ac54ad-kube-api-access-n89h5\") pod \"payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89\" (UID: \"c6b4fca1-447f-438a-a27d-2454c6ac54ad\") " pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" Oct 30 04:16:49.599987 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.599962 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4fca1-447f-438a-a27d-2454c6ac54ad-plugins-config-volume\") pod \"payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89\" (UID: \"c6b4fca1-447f-438a-a27d-2454c6ac54ad\") " pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" Oct 30 04:16:49.623631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.623586 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89h5\" (UniqueName: \"kubernetes.io/projected/c6b4fca1-447f-438a-a27d-2454c6ac54ad-kube-api-access-n89h5\") pod \"payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89\" (UID: \"c6b4fca1-447f-438a-a27d-2454c6ac54ad\") " pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" Oct 30 04:16:49.738682 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.738642 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" Oct 30 04:16:49.960213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:49.960184 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f"] Oct 30 04:16:49.962055 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:16:49.962026 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428462cc_ef9d_411b_a9fa_6681dc2f6540.slice/crio-aa4966289e93d7d893dd79db067fd02f204379f01af6233cddb4eef6af322f91 WatchSource:0}: Error finding container aa4966289e93d7d893dd79db067fd02f204379f01af6233cddb4eef6af322f91: Status 404 returned error can't find the container with id aa4966289e93d7d893dd79db067fd02f204379f01af6233cddb4eef6af322f91 Oct 30 04:16:50.068817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:50.068783 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89"] Oct 30 04:16:50.068952 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:16:50.068926 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b4fca1_447f_438a_a27d_2454c6ac54ad.slice/crio-64afe7353fe48c856c2e3088a751fdfd6717c80b74394523612f31f9e8679ac9 WatchSource:0}: Error finding container 64afe7353fe48c856c2e3088a751fdfd6717c80b74394523612f31f9e8679ac9: Status 404 returned error can't find the container with id 64afe7353fe48c856c2e3088a751fdfd6717c80b74394523612f31f9e8679ac9 Oct 30 04:16:50.446470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:50.446435 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" event={"ID":"c6b4fca1-447f-438a-a27d-2454c6ac54ad","Type":"ContainerStarted","Data":"45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415"} Oct 30 04:16:50.446911 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:50.446550 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" Oct 30 04:16:50.446911 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:50.446574 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" event={"ID":"c6b4fca1-447f-438a-a27d-2454c6ac54ad","Type":"ContainerStarted","Data":"64afe7353fe48c856c2e3088a751fdfd6717c80b74394523612f31f9e8679ac9"} Oct 30 04:16:50.448105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:50.448080 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" event={"ID":"428462cc-ef9d-411b-a9fa-6681dc2f6540","Type":"ContainerStarted","Data":"f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0"} Oct 30 04:16:50.448193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:50.448129 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" event={"ID":"428462cc-ef9d-411b-a9fa-6681dc2f6540","Type":"ContainerStarted","Data":"aa4966289e93d7d893dd79db067fd02f204379f01af6233cddb4eef6af322f91"} Oct 30 04:16:50.448337 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:50.448318 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" Oct 30 04:16:50.495190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:50.495140 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" podStartSLOduration=1.495125669 podStartE2EDuration="1.495125669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:50.49341075 +0000 UTC m=+1133.779285539" watchObservedRunningTime="2026-10-30 04:16:50.495125669 +0000 UTC m=+1133.781000443" Oct 30 04:16:50.538942 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:50.538898 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" podStartSLOduration=1.538885534 podStartE2EDuration="1.538885534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:50.536644469 +0000 UTC m=+1133.822519243" watchObservedRunningTime="2026-10-30 04:16:50.538885534 +0000 UTC m=+1133.824760309" Oct 30 04:16:50.793303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:50.793221 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:51.001530 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.001491 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:51.263812 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.263776 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:51.488200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.488164 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:51.870228 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.870194 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h"] Oct 30 04:16:51.874313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.874283 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:51.880521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.880486 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:51.906640 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.906613 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-cb1a18be-openshift-default-dockercfg-z2vsc\"" Oct 30 04:16:51.909142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.909116 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h"] Oct 30 04:16:51.932851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.932829 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4"] Oct 30 04:16:51.936603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.936578 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" Oct 30 04:16:51.953670 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.953649 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ad07bf73\"" Oct 30 04:16:51.958076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.958043 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4"] Oct 30 04:16:51.993638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:51.993613 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ad07bf73-dockercfg-7gsb5\"" Oct 30 04:16:52.035389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.033945 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dwk2\" (UniqueName: \"kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-kube-api-access-8dwk2\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.035389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.034011 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-istio-token\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.035389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.034051 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-envoy\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.035389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.034104 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba448995-8974-4206-88a6-2c4f0259759f-istiod-ca-cert\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.035389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.034138 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-credential-socket\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.035389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.034172 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3fa6fc-163a-483b-91b8-bce724c5254a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4\" (UID: \"ea3fa6fc-163a-483b-91b8-bce724c5254a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" Oct 30 04:16:52.035389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.034218 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrhcx\" (UniqueName: \"kubernetes.io/projected/ea3fa6fc-163a-483b-91b8-bce724c5254a-kube-api-access-xrhcx\") pod \"payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4\" (UID: \"ea3fa6fc-163a-483b-91b8-bce724c5254a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" Oct 30 04:16:52.035389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.034288 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba448995-8974-4206-88a6-2c4f0259759f-istio-podinfo\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.035389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.034323 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-data\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.035389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.034352 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-socket\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.035389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.034397 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-certs\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.046316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.039477 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz"] Oct 30 04:16:52.062447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.060820 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz"] Oct 30 04:16:52.062447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.060924 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" Oct 30 04:16:52.094873 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.094836 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:52.134988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.134924 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba448995-8974-4206-88a6-2c4f0259759f-istio-podinfo\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.134988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.134962 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-data\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.134988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.134982 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-socket\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.135249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135018 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-certs\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.135249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135047 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dwk2\" (UniqueName: \"kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-kube-api-access-8dwk2\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.135249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135077 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-istio-token\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.135249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135116 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-envoy\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.135249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135165 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba448995-8974-4206-88a6-2c4f0259759f-istiod-ca-cert\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.135249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135198 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-credential-socket\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.135249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135236 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3fa6fc-163a-483b-91b8-bce724c5254a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4\" (UID: \"ea3fa6fc-163a-483b-91b8-bce724c5254a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" Oct 30 04:16:52.135661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135281 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrhcx\" (UniqueName: \"kubernetes.io/projected/ea3fa6fc-163a-483b-91b8-bce724c5254a-kube-api-access-xrhcx\") pod \"payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4\" (UID: \"ea3fa6fc-163a-483b-91b8-bce724c5254a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" Oct 30 04:16:52.135661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135399 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-certs\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.135661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135578 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-socket\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.135819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135670 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-credential-socket\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.135819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135704 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-data\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.135970 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.135948 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba448995-8974-4206-88a6-2c4f0259759f-istiod-ca-cert\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.136073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.136049 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3fa6fc-163a-483b-91b8-bce724c5254a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4\" (UID: \"ea3fa6fc-163a-483b-91b8-bce724c5254a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" Oct 30 04:16:52.137751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.137725 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba448995-8974-4206-88a6-2c4f0259759f-istio-podinfo\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.138391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.138112 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-envoy\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.165756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.165704 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dwk2\" (UniqueName: \"kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-kube-api-access-8dwk2\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.166597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.166412 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrhcx\" (UniqueName: \"kubernetes.io/projected/ea3fa6fc-163a-483b-91b8-bce724c5254a-kube-api-access-xrhcx\") pod \"payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4\" (UID: \"ea3fa6fc-163a-483b-91b8-bce724c5254a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" Oct 30 04:16:52.171140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.171110 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-istio-token\") pod \"e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.190436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.190401 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:52.236147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.236114 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmxl4\" (UniqueName: \"kubernetes.io/projected/d7100422-dab2-4920-80ad-fd6b7fef342a-kube-api-access-tmxl4\") pod \"payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz\" (UID: \"d7100422-dab2-4920-80ad-fd6b7fef342a\") " pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" Oct 30 04:16:52.236290 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.236163 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7100422-dab2-4920-80ad-fd6b7fef342a-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz\" (UID: \"d7100422-dab2-4920-80ad-fd6b7fef342a\") " pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" Oct 30 04:16:52.252740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.252496 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" Oct 30 04:16:52.310880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.310843 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:52.337387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.337350 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmxl4\" (UniqueName: \"kubernetes.io/projected/d7100422-dab2-4920-80ad-fd6b7fef342a-kube-api-access-tmxl4\") pod \"payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz\" (UID: \"d7100422-dab2-4920-80ad-fd6b7fef342a\") " pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" Oct 30 04:16:52.337537 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.337450 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7100422-dab2-4920-80ad-fd6b7fef342a-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz\" (UID: \"d7100422-dab2-4920-80ad-fd6b7fef342a\") " pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" Oct 30 04:16:52.338569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.338541 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7100422-dab2-4920-80ad-fd6b7fef342a-plugins-config-volume\") pod \"payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz\" (UID: \"d7100422-dab2-4920-80ad-fd6b7fef342a\") " pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" Oct 30 04:16:52.363554 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.363526 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmxl4\" (UniqueName: \"kubernetes.io/projected/d7100422-dab2-4920-80ad-fd6b7fef342a-kube-api-access-tmxl4\") pod \"payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz\" (UID: \"d7100422-dab2-4920-80ad-fd6b7fef342a\") " pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" Oct 30 04:16:52.383331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.383306 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" Oct 30 04:16:52.517069 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:16:52.517040 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba448995_8974_4206_88a6_2c4f0259759f.slice/crio-ea069fa3c8fabbc3c142c3a9dc636bcc1e2ed10322e7e77542426cc8a50e8e8d WatchSource:0}: Error finding container ea069fa3c8fabbc3c142c3a9dc636bcc1e2ed10322e7e77542426cc8a50e8e8d: Status 404 returned error can't find the container with id ea069fa3c8fabbc3c142c3a9dc636bcc1e2ed10322e7e77542426cc8a50e8e8d Oct 30 04:16:52.517613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.517586 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h"] Oct 30 04:16:52.525095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.525062 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:16:52.525210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.525127 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:16:52.525210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.525158 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:16:52.573834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.573810 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4"] Oct 30 04:16:52.575166 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:16:52.575141 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3fa6fc_163a_483b_91b8_bce724c5254a.slice/crio-21f2d0516898dfb69a938c40d74385b4fe3fdb3ee41605205555bf2a726f8840 WatchSource:0}: Error finding container 21f2d0516898dfb69a938c40d74385b4fe3fdb3ee41605205555bf2a726f8840: Status 404 returned error can't find the container with id 21f2d0516898dfb69a938c40d74385b4fe3fdb3ee41605205555bf2a726f8840 Oct 30 04:16:52.702568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.702525 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz"] Oct 30 04:16:52.703553 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:16:52.703520 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7100422_dab2_4920_80ad_fd6b7fef342a.slice/crio-6a18772c51f20671866087180e9397b3141c5396fdbb373ea7e957bda11da422 WatchSource:0}: Error finding container 6a18772c51f20671866087180e9397b3141c5396fdbb373ea7e957bda11da422: Status 404 returned error can't find the container with id 6a18772c51f20671866087180e9397b3141c5396fdbb373ea7e957bda11da422 Oct 30 04:16:52.715494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:52.715455 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:53.485893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:53.485858 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" event={"ID":"ba448995-8974-4206-88a6-2c4f0259759f","Type":"ContainerStarted","Data":"88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939"} Oct 30 04:16:53.486066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:53.485919 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" event={"ID":"ba448995-8974-4206-88a6-2c4f0259759f","Type":"ContainerStarted","Data":"ea069fa3c8fabbc3c142c3a9dc636bcc1e2ed10322e7e77542426cc8a50e8e8d"} Oct 30 04:16:53.487548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:53.487518 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" event={"ID":"d7100422-dab2-4920-80ad-fd6b7fef342a","Type":"ContainerStarted","Data":"4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9"} Oct 30 04:16:53.487698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:53.487565 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" event={"ID":"d7100422-dab2-4920-80ad-fd6b7fef342a","Type":"ContainerStarted","Data":"6a18772c51f20671866087180e9397b3141c5396fdbb373ea7e957bda11da422"} Oct 30 04:16:53.487698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:53.487665 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" Oct 30 04:16:53.489111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:53.489092 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" event={"ID":"ea3fa6fc-163a-483b-91b8-bce724c5254a","Type":"ContainerStarted","Data":"2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762"} Oct 30 04:16:53.489213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:53.489123 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" event={"ID":"ea3fa6fc-163a-483b-91b8-bce724c5254a","Type":"ContainerStarted","Data":"21f2d0516898dfb69a938c40d74385b4fe3fdb3ee41605205555bf2a726f8840"} Oct 30 04:16:53.489286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:53.489231 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" Oct 30 04:16:53.533087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:53.533041 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" podStartSLOduration=2.5330277519999997 podStartE2EDuration="2.533027752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:53.53072836 +0000 UTC m=+1136.816603135" watchObservedRunningTime="2026-10-30 04:16:53.533027752 +0000 UTC m=+1136.818902527" Oct 30 04:16:53.572720 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:53.572674 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" podStartSLOduration=2.57266265 podStartE2EDuration="2.57266265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:53.570663429 +0000 UTC m=+1136.856538204" watchObservedRunningTime="2026-10-30 04:16:53.57266265 +0000 UTC m=+1136.858537425" Oct 30 04:16:53.611990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:53.611946 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" podStartSLOduration=2.611936333 podStartE2EDuration="2.611936333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:53.611123362 +0000 UTC m=+1136.896998150" watchObservedRunningTime="2026-10-30 04:16:53.611936333 +0000 UTC m=+1136.897811126" Oct 30 04:16:54.191084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:54.191048 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:54.197187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:54.197158 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:54.495281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:54.495198 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:54.496203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:54.496184 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:16:54.561797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:54.561760 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:56.373472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:56.373407 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:56.584197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:56.584151 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:57.428731 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.428690 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:57.454517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.454486 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n"] Oct 30 04:16:57.459356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.459339 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" Oct 30 04:16:57.480920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.480887 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n"] Oct 30 04:16:57.486799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.486775 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-cb1a18be\"" Oct 30 04:16:57.486799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.486791 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-cb1a18be-dockercfg-pdv77\"" Oct 30 04:16:57.515518 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.515493 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj"] Oct 30 04:16:57.520324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.520303 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" Oct 30 04:16:57.538331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.538304 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj"] Oct 30 04:16:57.614271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.614220 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj\" (UID: \"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" Oct 30 04:16:57.614497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.614291 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103c67b2-cc79-46fa-a2f4-43afa4553edb-plugins-config-volume\") pod \"payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n\" (UID: \"103c67b2-cc79-46fa-a2f4-43afa4553edb\") " pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" Oct 30 04:16:57.614497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.614321 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnflz\" (UniqueName: \"kubernetes.io/projected/103c67b2-cc79-46fa-a2f4-43afa4553edb-kube-api-access-lnflz\") pod \"payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n\" (UID: \"103c67b2-cc79-46fa-a2f4-43afa4553edb\") " pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" Oct 30 04:16:57.614497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.614349 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7sl\" (UniqueName: \"kubernetes.io/projected/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-kube-api-access-dm7sl\") pod \"payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj\" (UID: \"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" Oct 30 04:16:57.666394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.666358 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:57.715969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.715878 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj\" (UID: \"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" Oct 30 04:16:57.715969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.715935 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103c67b2-cc79-46fa-a2f4-43afa4553edb-plugins-config-volume\") pod \"payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n\" (UID: \"103c67b2-cc79-46fa-a2f4-43afa4553edb\") " pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" Oct 30 04:16:57.715969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.715954 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnflz\" (UniqueName: \"kubernetes.io/projected/103c67b2-cc79-46fa-a2f4-43afa4553edb-kube-api-access-lnflz\") pod \"payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n\" (UID: \"103c67b2-cc79-46fa-a2f4-43afa4553edb\") " pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" Oct 30 04:16:57.716181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.715973 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7sl\" (UniqueName: \"kubernetes.io/projected/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-kube-api-access-dm7sl\") pod \"payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj\" (UID: \"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" Oct 30 04:16:57.717045 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.717024 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj\" (UID: \"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" Oct 30 04:16:57.717317 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.717294 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103c67b2-cc79-46fa-a2f4-43afa4553edb-plugins-config-volume\") pod \"payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n\" (UID: \"103c67b2-cc79-46fa-a2f4-43afa4553edb\") " pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" Oct 30 04:16:57.739839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.739805 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7sl\" (UniqueName: \"kubernetes.io/projected/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-kube-api-access-dm7sl\") pod \"payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj\" (UID: \"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" Oct 30 04:16:57.739940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.739926 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnflz\" (UniqueName: \"kubernetes.io/projected/103c67b2-cc79-46fa-a2f4-43afa4553edb-kube-api-access-lnflz\") pod \"payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n\" (UID: \"103c67b2-cc79-46fa-a2f4-43afa4553edb\") " pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" Oct 30 04:16:57.772812 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.772788 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" Oct 30 04:16:57.838901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:57.833755 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" Oct 30 04:16:58.078064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:58.078033 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n"] Oct 30 04:16:58.078853 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:16:58.078822 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103c67b2_cc79_46fa_a2f4_43afa4553edb.slice/crio-0e35b52d88ca37e7fe3c601271c0f5a33d3634ecf515b4c6e84158a2ffea1c94 WatchSource:0}: Error finding container 0e35b52d88ca37e7fe3c601271c0f5a33d3634ecf515b4c6e84158a2ffea1c94: Status 404 returned error can't find the container with id 0e35b52d88ca37e7fe3c601271c0f5a33d3634ecf515b4c6e84158a2ffea1c94 Oct 30 04:16:58.155988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:58.155954 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj"] Oct 30 04:16:58.157356 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:16:58.157319 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452ddb1f_4bc8_4dcb_8dfc_2f0e6ff72f51.slice/crio-1844fde47effbc37d5c2ce4ea8c8672b27a5da5956f605f4150b2e60e48cdae1 WatchSource:0}: Error finding container 1844fde47effbc37d5c2ce4ea8c8672b27a5da5956f605f4150b2e60e48cdae1: Status 404 returned error can't find the container with id 1844fde47effbc37d5c2ce4ea8c8672b27a5da5956f605f4150b2e60e48cdae1 Oct 30 04:16:58.301746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:58.301708 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:16:58.431973 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:58.431945 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" Oct 30 04:16:58.524464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:58.524410 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" event={"ID":"103c67b2-cc79-46fa-a2f4-43afa4553edb","Type":"ContainerStarted","Data":"c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c"} Oct 30 04:16:58.524609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:58.524488 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" Oct 30 04:16:58.524609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:58.524509 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" event={"ID":"103c67b2-cc79-46fa-a2f4-43afa4553edb","Type":"ContainerStarted","Data":"0e35b52d88ca37e7fe3c601271c0f5a33d3634ecf515b4c6e84158a2ffea1c94"} Oct 30 04:16:58.526118 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:58.526091 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" event={"ID":"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51","Type":"ContainerStarted","Data":"c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9"} Oct 30 04:16:58.526249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:58.526131 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" event={"ID":"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51","Type":"ContainerStarted","Data":"1844fde47effbc37d5c2ce4ea8c8672b27a5da5956f605f4150b2e60e48cdae1"} Oct 30 04:16:58.526297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:58.526244 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" Oct 30 04:16:58.566875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:58.566817 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" podStartSLOduration=1.566800458 podStartE2EDuration="1.566800458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:58.565246231 +0000 UTC m=+1141.851121005" watchObservedRunningTime="2026-10-30 04:16:58.566800458 +0000 UTC m=+1141.852675234" Oct 30 04:16:58.608873 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:58.608826 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" podStartSLOduration=1.6088136739999999 podStartE2EDuration="1.608813674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:16:58.606137349 +0000 UTC m=+1141.892012124" watchObservedRunningTime="2026-10-30 04:16:58.608813674 +0000 UTC m=+1141.894688483" Oct 30 04:16:59.441068 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:16:59.441041 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" Oct 30 04:17:00.136412 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.136333 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:00.400193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.400102 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6"] Oct 30 04:17:00.404890 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.404865 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:00.423725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.423704 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6"] Oct 30 04:17:00.433688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.433659 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:17:00.433841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.433700 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:17:00.434076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.434035 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:17:00.434076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.434063 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:17:00.447761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.447729 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6\" (UID: \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:00.448076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.447834 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6\" (UID: \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:00.448076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.447911 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhjws\" (UniqueName: \"kubernetes.io/projected/f9901f22-d95d-4635-8a04-f8b573eb8f5d-kube-api-access-vhjws\") pod \"maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6\" (UID: \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:00.451006 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.450984 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" Oct 30 04:17:00.451105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.451051 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" Oct 30 04:17:00.548881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.548840 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhjws\" (UniqueName: \"kubernetes.io/projected/f9901f22-d95d-4635-8a04-f8b573eb8f5d-kube-api-access-vhjws\") pod \"maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6\" (UID: \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:00.549070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.548976 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6\" (UID: \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:00.549070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.549038 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6\" (UID: \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:00.549736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.549702 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6\" (UID: \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:00.551448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.551406 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6\" (UID: \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:00.589106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.589068 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhjws\" (UniqueName: \"kubernetes.io/projected/f9901f22-d95d-4635-8a04-f8b573eb8f5d-kube-api-access-vhjws\") pod \"maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6\" (UID: \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:00.723597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.723565 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:00.884486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.884452 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm"] Oct 30 04:17:00.889356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.889334 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:00.908454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.908041 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm"] Oct 30 04:17:00.979103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.978919 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-566c3724-69stm\" (UID: \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:00.979103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.978985 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-566c3724-69stm\" (UID: \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:00.979103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:00.979097 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xf7d\" (UniqueName: \"kubernetes.io/projected/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-kube-api-access-5xf7d\") pod \"maas-api-revoke-keys-e2e-migrate-566c3724-69stm\" (UID: \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:01.042283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.042246 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6"] Oct 30 04:17:01.042407 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:01.042305 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9901f22_d95d_4635_8a04_f8b573eb8f5d.slice/crio-6ea6c9cbaefe44007195d434aad37275074a12265730ed259818b4aa877345f0 WatchSource:0}: Error finding container 6ea6c9cbaefe44007195d434aad37275074a12265730ed259818b4aa877345f0: Status 404 returned error can't find the container with id 6ea6c9cbaefe44007195d434aad37275074a12265730ed259818b4aa877345f0 Oct 30 04:17:01.080341 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.080302 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-566c3724-69stm\" (UID: \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:01.080568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.080408 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-566c3724-69stm\" (UID: \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:01.080568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.080502 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xf7d\" (UniqueName: \"kubernetes.io/projected/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-kube-api-access-5xf7d\") pod \"maas-api-revoke-keys-e2e-migrate-566c3724-69stm\" (UID: \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:01.081197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.081167 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-566c3724-69stm\" (UID: \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:01.082743 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.082720 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-566c3724-69stm\" (UID: \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:01.104494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.104471 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xf7d\" (UniqueName: \"kubernetes.io/projected/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-kube-api-access-5xf7d\") pod \"maas-api-revoke-keys-e2e-migrate-566c3724-69stm\" (UID: \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:01.203645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.203618 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:01.480635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.480604 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" Oct 30 04:17:01.482224 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.482201 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" Oct 30 04:17:01.538957 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:01.538882 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ddb3ba_a8a4_4f27_adbc_812123db5d82.slice/crio-4f3016e62fb1e3e8510c2b2ddfc6f978909128ba4403aa1bacbbe4105e8d34ce WatchSource:0}: Error finding container 4f3016e62fb1e3e8510c2b2ddfc6f978909128ba4403aa1bacbbe4105e8d34ce: Status 404 returned error can't find the container with id 4f3016e62fb1e3e8510c2b2ddfc6f978909128ba4403aa1bacbbe4105e8d34ce Oct 30 04:17:01.539620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.539587 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm"] Oct 30 04:17:01.552139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.552111 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" event={"ID":"f9901f22-d95d-4635-8a04-f8b573eb8f5d","Type":"ContainerStarted","Data":"bb21beeb858af6fc28ab5a6ffd2c61f75bda28892411e903a82ea435f2a4b2c4"} Oct 30 04:17:01.552234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.552168 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" event={"ID":"f9901f22-d95d-4635-8a04-f8b573eb8f5d","Type":"ContainerStarted","Data":"6ea6c9cbaefe44007195d434aad37275074a12265730ed259818b4aa877345f0"} Oct 30 04:17:01.573118 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.573070 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" event={"ID":"a6ddb3ba-a8a4-4f27-adbc-812123db5d82","Type":"ContainerStarted","Data":"4f3016e62fb1e3e8510c2b2ddfc6f978909128ba4403aa1bacbbe4105e8d34ce"} Oct 30 04:17:01.623033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.622987 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" podStartSLOduration=1.6229738089999999 podStartE2EDuration="1.622973809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:01.62049877 +0000 UTC m=+1144.906373545" watchObservedRunningTime="2026-10-30 04:17:01.622973809 +0000 UTC m=+1144.908848584" Oct 30 04:17:01.949903 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:01.949868 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:02.589361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:02.589273 4793 generic.go:356] "Generic (PLEG): container finished" podID="f9901f22-d95d-4635-8a04-f8b573eb8f5d" containerID="bb21beeb858af6fc28ab5a6ffd2c61f75bda28892411e903a82ea435f2a4b2c4" exitCode=0 Oct 30 04:17:02.589862 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:02.589376 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" event={"ID":"f9901f22-d95d-4635-8a04-f8b573eb8f5d","Type":"ContainerDied","Data":"bb21beeb858af6fc28ab5a6ffd2c61f75bda28892411e903a82ea435f2a4b2c4"} Oct 30 04:17:02.591327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:02.591296 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" event={"ID":"a6ddb3ba-a8a4-4f27-adbc-812123db5d82","Type":"ContainerStarted","Data":"573d5caaea335baa8583d87694cd5fd29268f2dd92918b89d780741bf0fc7791"} Oct 30 04:17:02.686558 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:02.686502 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" podStartSLOduration=2.686483812 podStartE2EDuration="2.686483812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:02.672824787 +0000 UTC m=+1145.958699587" watchObservedRunningTime="2026-10-30 04:17:02.686483812 +0000 UTC m=+1145.972358587" Oct 30 04:17:03.599189 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:03.599147 4793 generic.go:356] "Generic (PLEG): container finished" podID="a6ddb3ba-a8a4-4f27-adbc-812123db5d82" containerID="573d5caaea335baa8583d87694cd5fd29268f2dd92918b89d780741bf0fc7791" exitCode=0 Oct 30 04:17:03.599587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:03.599237 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" event={"ID":"a6ddb3ba-a8a4-4f27-adbc-812123db5d82","Type":"ContainerDied","Data":"573d5caaea335baa8583d87694cd5fd29268f2dd92918b89d780741bf0fc7791"} Oct 30 04:17:03.790841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:03.790036 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:03.818302 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:03.818273 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:03.927644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:03.927614 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-xks-ca\") pod \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\" (UID: \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\") " Oct 30 04:17:03.927824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:03.927678 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9901f22-d95d-4635-8a04-f8b573eb8f5d-kube-api-access-vhjws\" (UniqueName: \"kubernetes.io/projected/f9901f22-d95d-4635-8a04-f8b573eb8f5d-kube-api-access-vhjws\") pod \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\" (UID: \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\") " Oct 30 04:17:03.927824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:03.927775 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-openshift-ca\") pod \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\" (UID: \"f9901f22-d95d-4635-8a04-f8b573eb8f5d\") " Oct 30 04:17:03.928048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:03.928027 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-openshift-ca" pod "f9901f22-d95d-4635-8a04-f8b573eb8f5d" (UID: "f9901f22-d95d-4635-8a04-f8b573eb8f5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:03.928232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:03.928213 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:03.929781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:03.929761 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9901f22-d95d-4635-8a04-f8b573eb8f5d-kube-api-access-vhjws" pod "f9901f22-d95d-4635-8a04-f8b573eb8f5d" (UID: "f9901f22-d95d-4635-8a04-f8b573eb8f5d"). InnerVolumeSpecName "kube-api-access-vhjws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:03.929847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:03.929825 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-xks-ca" pod "f9901f22-d95d-4635-8a04-f8b573eb8f5d" (UID: "f9901f22-d95d-4635-8a04-f8b573eb8f5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:04.029742 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.029715 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9901f22-d95d-4635-8a04-f8b573eb8f5d-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:04.029742 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.029740 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhjws\" (UniqueName: \"kubernetes.io/projected/f9901f22-d95d-4635-8a04-f8b573eb8f5d-kube-api-access-vhjws\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:04.140608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.140549 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:04.498272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.498242 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" Oct 30 04:17:04.498468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.498315 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" Oct 30 04:17:04.606741 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.606716 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" Oct 30 04:17:04.607170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.606749 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6" event={"ID":"f9901f22-d95d-4635-8a04-f8b573eb8f5d","Type":"ContainerDied","Data":"6ea6c9cbaefe44007195d434aad37275074a12265730ed259818b4aa877345f0"} Oct 30 04:17:04.607170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.606799 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ea6c9cbaefe44007195d434aad37275074a12265730ed259818b4aa877345f0" Oct 30 04:17:04.809265 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.809241 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:04.940536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.940503 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-openshift-ca\") pod \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\" (UID: \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\") " Oct 30 04:17:04.940725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.940558 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-kube-api-access-5xf7d\" (UniqueName: \"kubernetes.io/projected/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-kube-api-access-5xf7d\") pod \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\" (UID: \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\") " Oct 30 04:17:04.940725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.940590 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-xks-ca\") pod \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\" (UID: \"a6ddb3ba-a8a4-4f27-adbc-812123db5d82\") " Oct 30 04:17:04.940926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.940901 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-openshift-ca" pod "a6ddb3ba-a8a4-4f27-adbc-812123db5d82" (UID: "a6ddb3ba-a8a4-4f27-adbc-812123db5d82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:04.941037 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.941021 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:04.942687 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.942663 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-kube-api-access-5xf7d" pod "a6ddb3ba-a8a4-4f27-adbc-812123db5d82" (UID: "a6ddb3ba-a8a4-4f27-adbc-812123db5d82"). InnerVolumeSpecName "kube-api-access-5xf7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:04.942779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:04.942727 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-xks-ca" pod "a6ddb3ba-a8a4-4f27-adbc-812123db5d82" (UID: "a6ddb3ba-a8a4-4f27-adbc-812123db5d82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:05.042626 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.042541 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xf7d\" (UniqueName: \"kubernetes.io/projected/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-kube-api-access-5xf7d\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:05.042626 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.042579 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6ddb3ba-a8a4-4f27-adbc-812123db5d82-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:05.616889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.616855 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" Oct 30 04:17:05.617247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.616890 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm" event={"ID":"a6ddb3ba-a8a4-4f27-adbc-812123db5d82","Type":"ContainerDied","Data":"4f3016e62fb1e3e8510c2b2ddfc6f978909128ba4403aa1bacbbe4105e8d34ce"} Oct 30 04:17:05.617247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.616938 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f3016e62fb1e3e8510c2b2ddfc6f978909128ba4403aa1bacbbe4105e8d34ce" Oct 30 04:17:05.823309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.823217 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:05.948465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.948412 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf"] Oct 30 04:17:05.949185 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.949165 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a6ddb3ba-a8a4-4f27-adbc-812123db5d82" containerName="revoke-keys" Oct 30 04:17:05.949243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.949190 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ddb3ba-a8a4-4f27-adbc-812123db5d82" containerName="revoke-keys" Oct 30 04:17:05.949243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.949228 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f9901f22-d95d-4635-8a04-f8b573eb8f5d" containerName="revoke-keys" Oct 30 04:17:05.949243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.949236 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9901f22-d95d-4635-8a04-f8b573eb8f5d" containerName="revoke-keys" Oct 30 04:17:05.949363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.949351 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9901f22-d95d-4635-8a04-f8b573eb8f5d" containerName="revoke-keys" Oct 30 04:17:05.949394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.949371 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ddb3ba-a8a4-4f27-adbc-812123db5d82" containerName="revoke-keys" Oct 30 04:17:05.953341 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.953317 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:05.968730 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.968707 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:17:05.969449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.969429 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf"] Oct 30 04:17:05.978608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.978571 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:17:05.978608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.978574 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:17:05.978608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:05.978574 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:17:06.053507 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:06.053468 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-de5733f9-csflf\" (UID: \"defc4a49-1101-4ab5-8c59-54902ce57bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:06.053653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:06.053515 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhtj4\" (UniqueName: \"kubernetes.io/projected/defc4a49-1101-4ab5-8c59-54902ce57bcc-kube-api-access-jhtj4\") pod \"maas-api-revoke-keys-e2e-ait-de5733f9-csflf\" (UID: \"defc4a49-1101-4ab5-8c59-54902ce57bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:06.053740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:06.053717 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-de5733f9-csflf\" (UID: \"defc4a49-1101-4ab5-8c59-54902ce57bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:06.155181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:06.155147 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-de5733f9-csflf\" (UID: \"defc4a49-1101-4ab5-8c59-54902ce57bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:06.155354 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:06.155214 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-de5733f9-csflf\" (UID: \"defc4a49-1101-4ab5-8c59-54902ce57bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:06.155354 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:06.155253 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhtj4\" (UniqueName: \"kubernetes.io/projected/defc4a49-1101-4ab5-8c59-54902ce57bcc-kube-api-access-jhtj4\") pod \"maas-api-revoke-keys-e2e-ait-de5733f9-csflf\" (UID: \"defc4a49-1101-4ab5-8c59-54902ce57bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:06.156131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:06.156098 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-de5733f9-csflf\" (UID: \"defc4a49-1101-4ab5-8c59-54902ce57bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:06.157803 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:06.157774 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-de5733f9-csflf\" (UID: \"defc4a49-1101-4ab5-8c59-54902ce57bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:06.179427 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:06.179386 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhtj4\" (UniqueName: \"kubernetes.io/projected/defc4a49-1101-4ab5-8c59-54902ce57bcc-kube-api-access-jhtj4\") pod \"maas-api-revoke-keys-e2e-ait-de5733f9-csflf\" (UID: \"defc4a49-1101-4ab5-8c59-54902ce57bcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:06.270191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:06.270107 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:06.577318 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:06.577290 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf"] Oct 30 04:17:06.577794 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:06.577770 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefc4a49_1101_4ab5_8c59_54902ce57bcc.slice/crio-480dae6a84d2e6fd73e2c6bcee3379c72a46ec9fb33872e0e4c1bc69a7b604b4 WatchSource:0}: Error finding container 480dae6a84d2e6fd73e2c6bcee3379c72a46ec9fb33872e0e4c1bc69a7b604b4: Status 404 returned error can't find the container with id 480dae6a84d2e6fd73e2c6bcee3379c72a46ec9fb33872e0e4c1bc69a7b604b4 Oct 30 04:17:06.623811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:06.623783 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" event={"ID":"defc4a49-1101-4ab5-8c59-54902ce57bcc","Type":"ContainerStarted","Data":"480dae6a84d2e6fd73e2c6bcee3379c72a46ec9fb33872e0e4c1bc69a7b604b4"} Oct 30 04:17:07.633382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:07.633338 4793 generic.go:356] "Generic (PLEG): container finished" podID="defc4a49-1101-4ab5-8c59-54902ce57bcc" containerID="a041b1769079cf2907fe477363301aa6cc9321493653580b774dd8a25d00e85e" exitCode=0 Oct 30 04:17:07.633795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:07.633454 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" event={"ID":"defc4a49-1101-4ab5-8c59-54902ce57bcc","Type":"ContainerDied","Data":"a041b1769079cf2907fe477363301aa6cc9321493653580b774dd8a25d00e85e"} Oct 30 04:17:07.720153 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:07.720118 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:08.641944 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.641854 4793 generic.go:356] "Generic (PLEG): container finished" podID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerID="4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5" exitCode=0 Oct 30 04:17:08.641944 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.641933 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" event={"ID":"d8ab6828-4959-4412-9fdb-50237628cd8d","Type":"ContainerDied","Data":"4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5"} Oct 30 04:17:08.642746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.642447 4793 scope.go:122] "RemoveContainer" containerID="4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5" Oct 30 04:17:08.643756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.643721 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:08.643858 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.643790 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:08.643858 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.643820 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:08.823469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.823441 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:08.886183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.886147 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/defc4a49-1101-4ab5-8c59-54902ce57bcc-kube-api-access-jhtj4\" (UniqueName: \"kubernetes.io/projected/defc4a49-1101-4ab5-8c59-54902ce57bcc-kube-api-access-jhtj4\") pod \"defc4a49-1101-4ab5-8c59-54902ce57bcc\" (UID: \"defc4a49-1101-4ab5-8c59-54902ce57bcc\") " Oct 30 04:17:08.886364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.886198 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-openshift-ca\") pod \"defc4a49-1101-4ab5-8c59-54902ce57bcc\" (UID: \"defc4a49-1101-4ab5-8c59-54902ce57bcc\") " Oct 30 04:17:08.886364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.886313 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-xks-ca\") pod \"defc4a49-1101-4ab5-8c59-54902ce57bcc\" (UID: \"defc4a49-1101-4ab5-8c59-54902ce57bcc\") " Oct 30 04:17:08.886617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.886590 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-openshift-ca" pod "defc4a49-1101-4ab5-8c59-54902ce57bcc" (UID: "defc4a49-1101-4ab5-8c59-54902ce57bcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:08.888331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.888301 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defc4a49-1101-4ab5-8c59-54902ce57bcc-kube-api-access-jhtj4" pod "defc4a49-1101-4ab5-8c59-54902ce57bcc" (UID: "defc4a49-1101-4ab5-8c59-54902ce57bcc"). InnerVolumeSpecName "kube-api-access-jhtj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:08.888488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.888396 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-xks-ca" pod "defc4a49-1101-4ab5-8c59-54902ce57bcc" (UID: "defc4a49-1101-4ab5-8c59-54902ce57bcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:08.987951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.987920 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhtj4\" (UniqueName: \"kubernetes.io/projected/defc4a49-1101-4ab5-8c59-54902ce57bcc-kube-api-access-jhtj4\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:08.987951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.987947 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:08.987951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:08.987958 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/defc4a49-1101-4ab5-8c59-54902ce57bcc-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:09.534693 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:09.534619 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:09.538168 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:09.538143 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" Oct 30 04:17:09.538304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:09.538214 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" Oct 30 04:17:09.651748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:09.651093 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" Oct 30 04:17:09.651748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:09.651129 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf" event={"ID":"defc4a49-1101-4ab5-8c59-54902ce57bcc","Type":"ContainerDied","Data":"480dae6a84d2e6fd73e2c6bcee3379c72a46ec9fb33872e0e4c1bc69a7b604b4"} Oct 30 04:17:09.651748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:09.651184 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="480dae6a84d2e6fd73e2c6bcee3379c72a46ec9fb33872e0e4c1bc69a7b604b4" Oct 30 04:17:09.653123 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:09.653100 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" event={"ID":"d8ab6828-4959-4412-9fdb-50237628cd8d","Type":"ContainerStarted","Data":"6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585"} Oct 30 04:17:10.105455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:10.102716 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:17:10.105455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:10.103109 4793 patch_prober.go:29] interesting pod/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 04:17:10.105455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:10.103158 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 04:17:10.928984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:10.928946 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz"] Oct 30 04:17:10.987099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:10.986475 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh"] Oct 30 04:17:11.111436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:11.110838 4793 patch_prober.go:29] interesting pod/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 04:17:11.111436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:11.110902 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 04:17:11.189520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:11.189359 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:11.233495 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:11.233462 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:17:11.416169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:11.414757 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:11.906666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:11.905598 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89"] Oct 30 04:17:11.943180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:11.943148 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f"] Oct 30 04:17:12.104916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:12.104881 4793 patch_prober.go:29] interesting pod/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 04:17:12.121859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:12.121724 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 04:17:12.224461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:12.224025 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:12.447909 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:12.447878 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:13.032586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.032552 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:13.102837 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.102797 4793 patch_prober.go:29] interesting pod/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 04:17:13.103136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.103109 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 04:17:13.242288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.242262 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf"] Oct 30 04:17:13.242977 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.242953 4793 cpu_manager.go:414] "containerMap: removing container" podUID="defc4a49-1101-4ab5-8c59-54902ce57bcc" containerName="revoke-keys" Oct 30 04:17:13.242977 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.242976 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="defc4a49-1101-4ab5-8c59-54902ce57bcc" containerName="revoke-keys" Oct 30 04:17:13.243163 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.243074 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="defc4a49-1101-4ab5-8c59-54902ce57bcc" containerName="revoke-keys" Oct 30 04:17:13.246106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.246086 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:13.276439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.275684 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:17:13.276439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.275808 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:17:13.302490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.300469 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf"] Oct 30 04:17:13.323480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.323445 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:17:13.323645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.323630 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:17:13.378443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.378113 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxfj7\" (UniqueName: \"kubernetes.io/projected/28002b58-868c-4bcb-9e20-d4e2b4ef501a-kube-api-access-gxfj7\") pod \"maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf\" (UID: \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:13.378443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.378175 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf\" (UID: \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:13.378443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.378208 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf\" (UID: \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:13.489452 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.488097 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxfj7\" (UniqueName: \"kubernetes.io/projected/28002b58-868c-4bcb-9e20-d4e2b4ef501a-kube-api-access-gxfj7\") pod \"maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf\" (UID: \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:13.489452 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.488157 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf\" (UID: \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:13.489452 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.488189 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf\" (UID: \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:13.489754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.489592 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf\" (UID: \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:13.508449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.508235 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf\" (UID: \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:13.516571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.516547 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxfj7\" (UniqueName: \"kubernetes.io/projected/28002b58-868c-4bcb-9e20-d4e2b4ef501a-kube-api-access-gxfj7\") pod \"maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf\" (UID: \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:13.587492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.587389 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:13.921664 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:13.921634 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf"] Oct 30 04:17:14.108304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:14.107860 4793 patch_prober.go:29] interesting pod/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 04:17:14.108304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:14.107924 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 04:17:14.748595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:14.748547 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" event={"ID":"28002b58-868c-4bcb-9e20-d4e2b4ef501a","Type":"ContainerStarted","Data":"cf9c8e75be8e6aad1e7c009fe0dd0dc883c6f5528bfef409a85636b3ab6130cd"} Oct 30 04:17:14.748775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:14.748657 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" event={"ID":"28002b58-868c-4bcb-9e20-d4e2b4ef501a","Type":"ContainerStarted","Data":"3aa963fd0ab6fd2ba3f522bf23f147cc6f9c5777d578c7d7ec1508b6c679e04d"} Oct 30 04:17:14.792297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:14.792244 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" podStartSLOduration=1.7922309269999999 podStartE2EDuration="1.792230927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:14.789401816 +0000 UTC m=+1158.075276615" watchObservedRunningTime="2026-10-30 04:17:14.792230927 +0000 UTC m=+1158.078105702" Oct 30 04:17:15.107391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:15.107321 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:17:15.107788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:15.107754 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:17:15.108581 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:15.108562 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:17:15.116020 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:15.115994 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:15.756390 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:15.756353 4793 generic.go:356] "Generic (PLEG): container finished" podID="28002b58-868c-4bcb-9e20-d4e2b4ef501a" containerID="cf9c8e75be8e6aad1e7c009fe0dd0dc883c6f5528bfef409a85636b3ab6130cd" exitCode=0 Oct 30 04:17:15.756584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:15.756454 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" event={"ID":"28002b58-868c-4bcb-9e20-d4e2b4ef501a","Type":"ContainerDied","Data":"cf9c8e75be8e6aad1e7c009fe0dd0dc883c6f5528bfef409a85636b3ab6130cd"} Oct 30 04:17:15.929703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:15.929614 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" podUID="0863945a-4d72-4b2d-9c18-da05eb39bd48" containerName="payload-processing" containerID="cri-o://963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1" gracePeriod=25 Oct 30 04:17:15.987799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:15.987737 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" podUID="32651cda-6e75-4992-bfd2-92d9db399459" containerName="payload-pre-processing" containerID="cri-o://9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73" gracePeriod=25 Oct 30 04:17:16.265453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.265377 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" Oct 30 04:17:16.288744 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.288722 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" Oct 30 04:17:16.448666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.448632 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0863945a-4d72-4b2d-9c18-da05eb39bd48-kube-api-access-rrdtv\" (UniqueName: \"kubernetes.io/projected/0863945a-4d72-4b2d-9c18-da05eb39bd48-kube-api-access-rrdtv\") pod \"0863945a-4d72-4b2d-9c18-da05eb39bd48\" (UID: \"0863945a-4d72-4b2d-9c18-da05eb39bd48\") " Oct 30 04:17:16.448838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.448682 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0863945a-4d72-4b2d-9c18-da05eb39bd48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0863945a-4d72-4b2d-9c18-da05eb39bd48-plugins-config-volume\") pod \"0863945a-4d72-4b2d-9c18-da05eb39bd48\" (UID: \"0863945a-4d72-4b2d-9c18-da05eb39bd48\") " Oct 30 04:17:16.448838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.448821 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32651cda-6e75-4992-bfd2-92d9db399459-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32651cda-6e75-4992-bfd2-92d9db399459-plugins-config-volume\") pod \"32651cda-6e75-4992-bfd2-92d9db399459\" (UID: \"32651cda-6e75-4992-bfd2-92d9db399459\") " Oct 30 04:17:16.448962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.448873 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32651cda-6e75-4992-bfd2-92d9db399459-kube-api-access-cq9t7\" (UniqueName: \"kubernetes.io/projected/32651cda-6e75-4992-bfd2-92d9db399459-kube-api-access-cq9t7\") pod \"32651cda-6e75-4992-bfd2-92d9db399459\" (UID: \"32651cda-6e75-4992-bfd2-92d9db399459\") " Oct 30 04:17:16.449129 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.449101 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0863945a-4d72-4b2d-9c18-da05eb39bd48-plugins-config-volume" pod "0863945a-4d72-4b2d-9c18-da05eb39bd48" (UID: "0863945a-4d72-4b2d-9c18-da05eb39bd48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:16.449252 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.449217 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32651cda-6e75-4992-bfd2-92d9db399459-plugins-config-volume" pod "32651cda-6e75-4992-bfd2-92d9db399459" (UID: "32651cda-6e75-4992-bfd2-92d9db399459"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:16.449297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.449261 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0863945a-4d72-4b2d-9c18-da05eb39bd48-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:16.451054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.451035 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0863945a-4d72-4b2d-9c18-da05eb39bd48-kube-api-access-rrdtv" pod "0863945a-4d72-4b2d-9c18-da05eb39bd48" (UID: "0863945a-4d72-4b2d-9c18-da05eb39bd48"). InnerVolumeSpecName "kube-api-access-rrdtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:16.451128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.451052 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32651cda-6e75-4992-bfd2-92d9db399459-kube-api-access-cq9t7" pod "32651cda-6e75-4992-bfd2-92d9db399459" (UID: "32651cda-6e75-4992-bfd2-92d9db399459"). InnerVolumeSpecName "kube-api-access-cq9t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:16.549842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.549779 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrdtv\" (UniqueName: \"kubernetes.io/projected/0863945a-4d72-4b2d-9c18-da05eb39bd48-kube-api-access-rrdtv\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:16.549842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.549804 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32651cda-6e75-4992-bfd2-92d9db399459-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:16.549842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.549814 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9t7\" (UniqueName: \"kubernetes.io/projected/32651cda-6e75-4992-bfd2-92d9db399459-kube-api-access-cq9t7\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:16.713046 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.713003 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:16.763575 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.763543 4793 generic.go:356] "Generic (PLEG): container finished" podID="32651cda-6e75-4992-bfd2-92d9db399459" containerID="9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73" exitCode=0 Oct 30 04:17:16.763746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.763576 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" event={"ID":"32651cda-6e75-4992-bfd2-92d9db399459","Type":"ContainerDied","Data":"9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73"} Oct 30 04:17:16.763746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.763598 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" Oct 30 04:17:16.763746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.763643 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh" event={"ID":"32651cda-6e75-4992-bfd2-92d9db399459","Type":"ContainerDied","Data":"b21222ee60a30a50f2ac6848b918a3403bfb2378c93c859d81840d8d9ce91f59"} Oct 30 04:17:16.763746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.763685 4793 scope.go:122] "RemoveContainer" containerID="9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73" Oct 30 04:17:16.765016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.764991 4793 generic.go:356] "Generic (PLEG): container finished" podID="0863945a-4d72-4b2d-9c18-da05eb39bd48" containerID="963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1" exitCode=0 Oct 30 04:17:16.765160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.765044 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" Oct 30 04:17:16.765160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.765064 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" event={"ID":"0863945a-4d72-4b2d-9c18-da05eb39bd48","Type":"ContainerDied","Data":"963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1"} Oct 30 04:17:16.765160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.765112 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz" event={"ID":"0863945a-4d72-4b2d-9c18-da05eb39bd48","Type":"ContainerDied","Data":"586f989d6c18eeb76bb1bc8f699367e3bf9d7dbaf31beb68d77af29805353089"} Oct 30 04:17:16.775793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.775770 4793 scope.go:122] "RemoveContainer" containerID="9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73" Oct 30 04:17:16.776196 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:16.776167 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73\": container with ID starting with 9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73 not found: ID does not exist" containerID="9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73" Oct 30 04:17:16.776300 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.776206 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73"} err="failed to get container status \"9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73\": rpc error: code = NotFound desc = could not find container \"9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73\": container with ID starting with 9491e88244c798c5bafeec820fe470edf55845a9b781ec0896f195a15bf1ef73 not found: ID does not exist" Oct 30 04:17:16.776300 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.776231 4793 scope.go:122] "RemoveContainer" containerID="963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1" Oct 30 04:17:16.785608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.785589 4793 scope.go:122] "RemoveContainer" containerID="963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1" Oct 30 04:17:16.785926 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:16.785901 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1\": container with ID starting with 963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1 not found: ID does not exist" containerID="963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1" Oct 30 04:17:16.785998 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.785935 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1"} err="failed to get container status \"963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1\": rpc error: code = NotFound desc = could not find container \"963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1\": container with ID starting with 963c0a57d57a3da05ba869d4184e6128d8f4a730ed99a0e33d955136e846e2c1 not found: ID does not exist" Oct 30 04:17:16.820615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.820533 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz"] Oct 30 04:17:16.832328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.832304 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-eb861bd9-5b6494876d-zwftz"] Oct 30 04:17:16.867867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.867683 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0863945a-4d72-4b2d-9c18-da05eb39bd48" path="/var/lib/kubelet/pods/0863945a-4d72-4b2d-9c18-da05eb39bd48/volumes" Oct 30 04:17:16.868284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.868260 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh"] Oct 30 04:17:16.872337 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.872310 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-eb861bd9-989d48df5-j2fzh"] Oct 30 04:17:16.906146 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.906116 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" podUID="c6b4fca1-447f-438a-a27d-2454c6ac54ad" containerName="payload-processing" containerID="cri-o://45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415" gracePeriod=25 Oct 30 04:17:16.911363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.911342 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:16.944021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:16.943973 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" podUID="428462cc-ef9d-411b-a9fa-6681dc2f6540" containerName="payload-pre-processing" containerID="cri-o://f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0" gracePeriod=25 Oct 30 04:17:17.056690 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.056660 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-openshift-ca\") pod \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\" (UID: \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\") " Oct 30 04:17:17.056797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.056755 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28002b58-868c-4bcb-9e20-d4e2b4ef501a-kube-api-access-gxfj7\" (UniqueName: \"kubernetes.io/projected/28002b58-868c-4bcb-9e20-d4e2b4ef501a-kube-api-access-gxfj7\") pod \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\" (UID: \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\") " Oct 30 04:17:17.056962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.056942 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-xks-ca\") pod \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\" (UID: \"28002b58-868c-4bcb-9e20-d4e2b4ef501a\") " Oct 30 04:17:17.057089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.057056 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-openshift-ca" pod "28002b58-868c-4bcb-9e20-d4e2b4ef501a" (UID: "28002b58-868c-4bcb-9e20-d4e2b4ef501a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:17.057342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.057318 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:17.058934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.058911 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28002b58-868c-4bcb-9e20-d4e2b4ef501a-kube-api-access-gxfj7" pod "28002b58-868c-4bcb-9e20-d4e2b4ef501a" (UID: "28002b58-868c-4bcb-9e20-d4e2b4ef501a"). InnerVolumeSpecName "kube-api-access-gxfj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:17.059285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.059267 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-xks-ca" pod "28002b58-868c-4bcb-9e20-d4e2b4ef501a" (UID: "28002b58-868c-4bcb-9e20-d4e2b4ef501a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:17.170586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.160929 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28002b58-868c-4bcb-9e20-d4e2b4ef501a-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:17.170586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.160956 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxfj7\" (UniqueName: \"kubernetes.io/projected/28002b58-868c-4bcb-9e20-d4e2b4ef501a-kube-api-access-gxfj7\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:17.205611 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.205582 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt"] Oct 30 04:17:17.213584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.213506 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" Oct 30 04:17:17.216630 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.216612 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" Oct 30 04:17:17.239803 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.239770 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx"] Oct 30 04:17:17.362635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.362552 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b4fca1-447f-438a-a27d-2454c6ac54ad-kube-api-access-n89h5\" (UniqueName: \"kubernetes.io/projected/c6b4fca1-447f-438a-a27d-2454c6ac54ad-kube-api-access-n89h5\") pod \"c6b4fca1-447f-438a-a27d-2454c6ac54ad\" (UID: \"c6b4fca1-447f-438a-a27d-2454c6ac54ad\") " Oct 30 04:17:17.363101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.362634 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/428462cc-ef9d-411b-a9fa-6681dc2f6540-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428462cc-ef9d-411b-a9fa-6681dc2f6540-plugins-config-volume\") pod \"428462cc-ef9d-411b-a9fa-6681dc2f6540\" (UID: \"428462cc-ef9d-411b-a9fa-6681dc2f6540\") " Oct 30 04:17:17.363101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.362717 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b4fca1-447f-438a-a27d-2454c6ac54ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4fca1-447f-438a-a27d-2454c6ac54ad-plugins-config-volume\") pod \"c6b4fca1-447f-438a-a27d-2454c6ac54ad\" (UID: \"c6b4fca1-447f-438a-a27d-2454c6ac54ad\") " Oct 30 04:17:17.363101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.362793 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/428462cc-ef9d-411b-a9fa-6681dc2f6540-kube-api-access-fqrpm\" (UniqueName: \"kubernetes.io/projected/428462cc-ef9d-411b-a9fa-6681dc2f6540-kube-api-access-fqrpm\") pod \"428462cc-ef9d-411b-a9fa-6681dc2f6540\" (UID: \"428462cc-ef9d-411b-a9fa-6681dc2f6540\") " Oct 30 04:17:17.363101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.363041 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428462cc-ef9d-411b-a9fa-6681dc2f6540-plugins-config-volume" pod "428462cc-ef9d-411b-a9fa-6681dc2f6540" (UID: "428462cc-ef9d-411b-a9fa-6681dc2f6540"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:17.363331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.363119 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b4fca1-447f-438a-a27d-2454c6ac54ad-plugins-config-volume" pod "c6b4fca1-447f-438a-a27d-2454c6ac54ad" (UID: "c6b4fca1-447f-438a-a27d-2454c6ac54ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:17.363331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.363140 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428462cc-ef9d-411b-a9fa-6681dc2f6540-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:17.364810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.364784 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b4fca1-447f-438a-a27d-2454c6ac54ad-kube-api-access-n89h5" pod "c6b4fca1-447f-438a-a27d-2454c6ac54ad" (UID: "c6b4fca1-447f-438a-a27d-2454c6ac54ad"). InnerVolumeSpecName "kube-api-access-n89h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:17.364947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.364927 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428462cc-ef9d-411b-a9fa-6681dc2f6540-kube-api-access-fqrpm" pod "428462cc-ef9d-411b-a9fa-6681dc2f6540" (UID: "428462cc-ef9d-411b-a9fa-6681dc2f6540"). InnerVolumeSpecName "kube-api-access-fqrpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:17.452469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.450995 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:17.464386 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.464349 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b4fca1-447f-438a-a27d-2454c6ac54ad-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:17.464386 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.464387 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqrpm\" (UniqueName: \"kubernetes.io/projected/428462cc-ef9d-411b-a9fa-6681dc2f6540-kube-api-access-fqrpm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:17.464628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.464405 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n89h5\" (UniqueName: \"kubernetes.io/projected/c6b4fca1-447f-438a-a27d-2454c6ac54ad-kube-api-access-n89h5\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:17.695480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.695446 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:17.773484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.773452 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" Oct 30 04:17:17.773659 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.773449 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf" event={"ID":"28002b58-868c-4bcb-9e20-d4e2b4ef501a","Type":"ContainerDied","Data":"3aa963fd0ab6fd2ba3f522bf23f147cc6f9c5777d578c7d7ec1508b6c679e04d"} Oct 30 04:17:17.773659 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.773625 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3aa963fd0ab6fd2ba3f522bf23f147cc6f9c5777d578c7d7ec1508b6c679e04d" Oct 30 04:17:17.775160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.775127 4793 generic.go:356] "Generic (PLEG): container finished" podID="428462cc-ef9d-411b-a9fa-6681dc2f6540" containerID="f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0" exitCode=0 Oct 30 04:17:17.775360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.775222 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" Oct 30 04:17:17.775360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.775258 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" event={"ID":"428462cc-ef9d-411b-a9fa-6681dc2f6540","Type":"ContainerDied","Data":"f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0"} Oct 30 04:17:17.775360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.775305 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f" event={"ID":"428462cc-ef9d-411b-a9fa-6681dc2f6540","Type":"ContainerDied","Data":"aa4966289e93d7d893dd79db067fd02f204379f01af6233cddb4eef6af322f91"} Oct 30 04:17:17.775360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.775327 4793 scope.go:122] "RemoveContainer" containerID="f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0" Oct 30 04:17:17.778782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.778738 4793 generic.go:356] "Generic (PLEG): container finished" podID="c6b4fca1-447f-438a-a27d-2454c6ac54ad" containerID="45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415" exitCode=0 Oct 30 04:17:17.778876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.778796 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" event={"ID":"c6b4fca1-447f-438a-a27d-2454c6ac54ad","Type":"ContainerDied","Data":"45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415"} Oct 30 04:17:17.778876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.778842 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" event={"ID":"c6b4fca1-447f-438a-a27d-2454c6ac54ad","Type":"ContainerDied","Data":"64afe7353fe48c856c2e3088a751fdfd6717c80b74394523612f31f9e8679ac9"} Oct 30 04:17:17.779005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.778915 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89" Oct 30 04:17:17.786888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.786870 4793 scope.go:122] "RemoveContainer" containerID="f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0" Oct 30 04:17:17.787159 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:17.787133 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0\": container with ID starting with f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0 not found: ID does not exist" containerID="f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0" Oct 30 04:17:17.787306 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.787170 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0"} err="failed to get container status \"f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0\": rpc error: code = NotFound desc = could not find container \"f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0\": container with ID starting with f8236e7992101d2d4599adbffdf1eb417657a31c7a6a399f0879d24d771873c0 not found: ID does not exist" Oct 30 04:17:17.787306 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.787194 4793 scope.go:122] "RemoveContainer" containerID="45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415" Oct 30 04:17:17.798216 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.798196 4793 scope.go:122] "RemoveContainer" containerID="45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415" Oct 30 04:17:17.798490 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:17.798464 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415\": container with ID starting with 45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415 not found: ID does not exist" containerID="45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415" Oct 30 04:17:17.798586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.798500 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415"} err="failed to get container status \"45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415\": rpc error: code = NotFound desc = could not find container \"45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415\": container with ID starting with 45624047a9838e55b17eb805dcb045f12aad8589adbd2bf4991ffa62e4d7b415 not found: ID does not exist" Oct 30 04:17:17.933433 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.932506 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f"] Oct 30 04:17:17.935508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.935477 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-566c3724-866768ccdc-j265f"] Oct 30 04:17:17.947451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.947361 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89"] Oct 30 04:17:17.951679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:17.951646 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-566c3724-99f7c99b-nnf89"] Oct 30 04:17:18.263126 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.263100 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:18.628693 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.628601 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:18.771161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.771131 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6"] Oct 30 04:17:18.783518 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.783493 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-eb861bd9-f6js6"] Oct 30 04:17:18.795036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.795011 4793 generic.go:356] "Generic (PLEG): container finished" podID="ba448995-8974-4206-88a6-2c4f0259759f" containerID="88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939" exitCode=0 Oct 30 04:17:18.795480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.795102 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" event={"ID":"ba448995-8974-4206-88a6-2c4f0259759f","Type":"ContainerDied","Data":"88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939"} Oct 30 04:17:18.795627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.795609 4793 scope.go:122] "RemoveContainer" containerID="88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939" Oct 30 04:17:18.797138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.797103 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:18.797250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.797185 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:18.797250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.797233 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:18.864200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.857879 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32651cda-6e75-4992-bfd2-92d9db399459" path="/var/lib/kubelet/pods/32651cda-6e75-4992-bfd2-92d9db399459/volumes" Oct 30 04:17:18.864200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.858370 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="428462cc-ef9d-411b-a9fa-6681dc2f6540" path="/var/lib/kubelet/pods/428462cc-ef9d-411b-a9fa-6681dc2f6540/volumes" Oct 30 04:17:18.864200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.858923 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b4fca1-447f-438a-a27d-2454c6ac54ad" path="/var/lib/kubelet/pods/c6b4fca1-447f-438a-a27d-2454c6ac54ad/volumes" Oct 30 04:17:18.864200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.862330 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9901f22-d95d-4635-8a04-f8b573eb8f5d" path="/var/lib/kubelet/pods/f9901f22-d95d-4635-8a04-f8b573eb8f5d/volumes" Oct 30 04:17:18.871852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:18.871829 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:19.097072 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:19.097037 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:19.221783 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:19.221753 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm"] Oct 30 04:17:19.224679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:19.224656 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-566c3724-69stm"] Oct 30 04:17:19.302445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:19.301823 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:19.510497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:19.509055 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:19.644511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:19.644479 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf"] Oct 30 04:17:19.657442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:19.655761 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-de5733f9-csflf"] Oct 30 04:17:19.724145 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:19.724110 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:19.805286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:19.805203 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" event={"ID":"ba448995-8974-4206-88a6-2c4f0259759f","Type":"ContainerStarted","Data":"da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8"} Oct 30 04:17:19.926890 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:19.926854 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:20.191098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.190888 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:17:20.192166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.192137 4793 patch_prober.go:29] interesting pod/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 04:17:20.192307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.192197 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 04:17:20.342732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.342700 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s"] Oct 30 04:17:20.343138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343126 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c6b4fca1-447f-438a-a27d-2454c6ac54ad" containerName="payload-processing" Oct 30 04:17:20.343182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343139 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b4fca1-447f-438a-a27d-2454c6ac54ad" containerName="payload-processing" Oct 30 04:17:20.343182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343150 4793 cpu_manager.go:414] "containerMap: removing container" podUID="428462cc-ef9d-411b-a9fa-6681dc2f6540" containerName="payload-pre-processing" Oct 30 04:17:20.343182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343155 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="428462cc-ef9d-411b-a9fa-6681dc2f6540" containerName="payload-pre-processing" Oct 30 04:17:20.343182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343166 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0863945a-4d72-4b2d-9c18-da05eb39bd48" containerName="payload-processing" Oct 30 04:17:20.343182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343172 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0863945a-4d72-4b2d-9c18-da05eb39bd48" containerName="payload-processing" Oct 30 04:17:20.343382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343188 4793 cpu_manager.go:414] "containerMap: removing container" podUID="28002b58-868c-4bcb-9e20-d4e2b4ef501a" containerName="revoke-keys" Oct 30 04:17:20.343382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343193 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28002b58-868c-4bcb-9e20-d4e2b4ef501a" containerName="revoke-keys" Oct 30 04:17:20.343382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343200 4793 cpu_manager.go:414] "containerMap: removing container" podUID="32651cda-6e75-4992-bfd2-92d9db399459" containerName="payload-pre-processing" Oct 30 04:17:20.343382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343204 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32651cda-6e75-4992-bfd2-92d9db399459" containerName="payload-pre-processing" Oct 30 04:17:20.343382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343272 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="28002b58-868c-4bcb-9e20-d4e2b4ef501a" containerName="revoke-keys" Oct 30 04:17:20.343382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343280 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="32651cda-6e75-4992-bfd2-92d9db399459" containerName="payload-pre-processing" Oct 30 04:17:20.343382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343288 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b4fca1-447f-438a-a27d-2454c6ac54ad" containerName="payload-processing" Oct 30 04:17:20.343382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343293 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="428462cc-ef9d-411b-a9fa-6681dc2f6540" containerName="payload-pre-processing" Oct 30 04:17:20.343382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.343298 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="0863945a-4d72-4b2d-9c18-da05eb39bd48" containerName="payload-processing" Oct 30 04:17:20.346302 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.346281 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:20.348041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.348011 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:20.384203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.383380 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:17:20.384203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.383531 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:17:20.384203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.384150 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:17:20.397877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.397318 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s"] Oct 30 04:17:20.397877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.397584 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:17:20.422447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.418991 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s\" (UID: \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:20.422447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.419113 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s\" (UID: \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:20.422447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.419167 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsvnd\" (UniqueName: \"kubernetes.io/projected/daa5f3fc-17e8-42ae-a04a-7e09c1894407-kube-api-access-tsvnd\") pod \"maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s\" (UID: \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:20.519988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.519900 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s\" (UID: \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:20.519988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.519965 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s\" (UID: \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:20.520195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.519993 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsvnd\" (UniqueName: \"kubernetes.io/projected/daa5f3fc-17e8-42ae-a04a-7e09c1894407-kube-api-access-tsvnd\") pod \"maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s\" (UID: \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:20.520732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.520704 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s\" (UID: \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:20.522207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.522190 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s\" (UID: \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:20.543603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.543577 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsvnd\" (UniqueName: \"kubernetes.io/projected/daa5f3fc-17e8-42ae-a04a-7e09c1894407-kube-api-access-tsvnd\") pod \"maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s\" (UID: \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:20.544481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.544456 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:20.720005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.719975 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:20.848063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.848027 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ddb3ba-a8a4-4f27-adbc-812123db5d82" path="/var/lib/kubelet/pods/a6ddb3ba-a8a4-4f27-adbc-812123db5d82/volumes" Oct 30 04:17:20.848650 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.848627 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="defc4a49-1101-4ab5-8c59-54902ce57bcc" path="/var/lib/kubelet/pods/defc4a49-1101-4ab5-8c59-54902ce57bcc/volumes" Oct 30 04:17:20.877441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:20.876846 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:21.096779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:21.096751 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s"] Oct 30 04:17:21.096779 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:21.096758 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa5f3fc_17e8_42ae_a04a_7e09c1894407.slice/crio-2a0c73ae9a9758d3d55808e1b818dd1834d59f44f3ca5a6c9b1ed34de3c82073 WatchSource:0}: Error finding container 2a0c73ae9a9758d3d55808e1b818dd1834d59f44f3ca5a6c9b1ed34de3c82073: Status 404 returned error can't find the container with id 2a0c73ae9a9758d3d55808e1b818dd1834d59f44f3ca5a6c9b1ed34de3c82073 Oct 30 04:17:21.139643 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:21.139606 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:21.190886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:21.190857 4793 patch_prober.go:29] interesting pod/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 04:17:21.191042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:21.190906 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 04:17:21.419312 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:21.418625 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:21.814701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:21.814668 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:21.826277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:21.826244 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" event={"ID":"daa5f3fc-17e8-42ae-a04a-7e09c1894407","Type":"ContainerStarted","Data":"262f29933a9df8db0cc4376515785dee3123bd56dc2550a4c916b874e688176a"} Oct 30 04:17:21.826406 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:21.826305 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" event={"ID":"daa5f3fc-17e8-42ae-a04a-7e09c1894407","Type":"ContainerStarted","Data":"2a0c73ae9a9758d3d55808e1b818dd1834d59f44f3ca5a6c9b1ed34de3c82073"} Oct 30 04:17:21.881439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:21.881228 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" podStartSLOduration=1.881212452 podStartE2EDuration="1.881212452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:21.88068526 +0000 UTC m=+1165.166560035" watchObservedRunningTime="2026-10-30 04:17:21.881212452 +0000 UTC m=+1165.167087229" Oct 30 04:17:22.191665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.191636 4793 patch_prober.go:29] interesting pod/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 04:17:22.191820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.191695 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 04:17:22.206898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.206843 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" podUID="9b8a8f22-0902-4338-8991-7eaeeb963120" containerName="payload-processing" containerID="cri-o://51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb" gracePeriod=25 Oct 30 04:17:22.241022 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.240989 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" podUID="ab80769a-d645-4f51-bc10-f22ea546f798" containerName="payload-pre-processing" containerID="cri-o://c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60" gracePeriod=25 Oct 30 04:17:22.331054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.331016 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:22.530984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.530962 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" Oct 30 04:17:22.609310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.609281 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" Oct 30 04:17:22.644965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.644939 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8a8f22-0902-4338-8991-7eaeeb963120-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8a8f22-0902-4338-8991-7eaeeb963120-plugins-config-volume\") pod \"9b8a8f22-0902-4338-8991-7eaeeb963120\" (UID: \"9b8a8f22-0902-4338-8991-7eaeeb963120\") " Oct 30 04:17:22.645106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.645005 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab80769a-d645-4f51-bc10-f22ea546f798-kube-api-access-5z6rr\" (UniqueName: \"kubernetes.io/projected/ab80769a-d645-4f51-bc10-f22ea546f798-kube-api-access-5z6rr\") pod \"ab80769a-d645-4f51-bc10-f22ea546f798\" (UID: \"ab80769a-d645-4f51-bc10-f22ea546f798\") " Oct 30 04:17:22.645106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.645039 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab80769a-d645-4f51-bc10-f22ea546f798-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab80769a-d645-4f51-bc10-f22ea546f798-plugins-config-volume\") pod \"ab80769a-d645-4f51-bc10-f22ea546f798\" (UID: \"ab80769a-d645-4f51-bc10-f22ea546f798\") " Oct 30 04:17:22.645106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.645058 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8a8f22-0902-4338-8991-7eaeeb963120-kube-api-access-jhgs7\" (UniqueName: \"kubernetes.io/projected/9b8a8f22-0902-4338-8991-7eaeeb963120-kube-api-access-jhgs7\") pod \"9b8a8f22-0902-4338-8991-7eaeeb963120\" (UID: \"9b8a8f22-0902-4338-8991-7eaeeb963120\") " Oct 30 04:17:22.645332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.645309 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8a8f22-0902-4338-8991-7eaeeb963120-plugins-config-volume" pod "9b8a8f22-0902-4338-8991-7eaeeb963120" (UID: "9b8a8f22-0902-4338-8991-7eaeeb963120"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:22.645482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.645459 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab80769a-d645-4f51-bc10-f22ea546f798-plugins-config-volume" pod "ab80769a-d645-4f51-bc10-f22ea546f798" (UID: "ab80769a-d645-4f51-bc10-f22ea546f798"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:22.647245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.647218 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab80769a-d645-4f51-bc10-f22ea546f798-kube-api-access-5z6rr" pod "ab80769a-d645-4f51-bc10-f22ea546f798" (UID: "ab80769a-d645-4f51-bc10-f22ea546f798"). InnerVolumeSpecName "kube-api-access-5z6rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:22.647347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.647238 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8a8f22-0902-4338-8991-7eaeeb963120-kube-api-access-jhgs7" pod "9b8a8f22-0902-4338-8991-7eaeeb963120" (UID: "9b8a8f22-0902-4338-8991-7eaeeb963120"). InnerVolumeSpecName "kube-api-access-jhgs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:22.747809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.747715 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab80769a-d645-4f51-bc10-f22ea546f798-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:22.747809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.747750 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhgs7\" (UniqueName: \"kubernetes.io/projected/9b8a8f22-0902-4338-8991-7eaeeb963120-kube-api-access-jhgs7\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:22.747809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.747766 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b8a8f22-0902-4338-8991-7eaeeb963120-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:22.747809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.747781 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z6rr\" (UniqueName: \"kubernetes.io/projected/ab80769a-d645-4f51-bc10-f22ea546f798-kube-api-access-5z6rr\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:22.813284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.813247 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r"] Oct 30 04:17:22.813901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.813877 4793 cpu_manager.go:414] "containerMap: removing container" podUID="9b8a8f22-0902-4338-8991-7eaeeb963120" containerName="payload-processing" Oct 30 04:17:22.813901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.813901 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8a8f22-0902-4338-8991-7eaeeb963120" containerName="payload-processing" Oct 30 04:17:22.814084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.813925 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ab80769a-d645-4f51-bc10-f22ea546f798" containerName="payload-pre-processing" Oct 30 04:17:22.814084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.813931 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab80769a-d645-4f51-bc10-f22ea546f798" containerName="payload-pre-processing" Oct 30 04:17:22.814084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.814006 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab80769a-d645-4f51-bc10-f22ea546f798" containerName="payload-pre-processing" Oct 30 04:17:22.814084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.814020 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8a8f22-0902-4338-8991-7eaeeb963120" containerName="payload-processing" Oct 30 04:17:22.816775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.816755 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.838465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.838440 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-a2f984-openshift-default-dockercfg-pqhlj\"" Oct 30 04:17:22.845984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.845956 4793 generic.go:356] "Generic (PLEG): container finished" podID="9b8a8f22-0902-4338-8991-7eaeeb963120" containerID="51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb" exitCode=0 Oct 30 04:17:22.846259 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.846131 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" Oct 30 04:17:22.848878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.848852 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r"] Oct 30 04:17:22.848982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.848908 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" event={"ID":"9b8a8f22-0902-4338-8991-7eaeeb963120","Type":"ContainerDied","Data":"51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb"} Oct 30 04:17:22.848982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.848941 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt" event={"ID":"9b8a8f22-0902-4338-8991-7eaeeb963120","Type":"ContainerDied","Data":"e83632a1646c30c5fdc659c00dfa0c5616a4b904637563213efe8cbf67635449"} Oct 30 04:17:22.849359 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.849205 4793 scope.go:122] "RemoveContainer" containerID="51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb" Oct 30 04:17:22.850096 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.850070 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-data\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.850268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.850248 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da898aa-742b-41a7-9c66-ab97460daf99-istiod-ca-cert\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.850498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.850479 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-certs\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.850664 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.850640 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-credential-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.850757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.850691 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.850757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.850743 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-envoy\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.850864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.850790 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da898aa-742b-41a7-9c66-ab97460daf99-istio-podinfo\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.850864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.850826 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-istio-token\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.851036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.850863 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw957\" (UniqueName: \"kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-kube-api-access-hw957\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.851716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.851696 4793 generic.go:356] "Generic (PLEG): container finished" podID="daa5f3fc-17e8-42ae-a04a-7e09c1894407" containerID="262f29933a9df8db0cc4376515785dee3123bd56dc2550a4c916b874e688176a" exitCode=0 Oct 30 04:17:22.851959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.851939 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" event={"ID":"daa5f3fc-17e8-42ae-a04a-7e09c1894407","Type":"ContainerDied","Data":"262f29933a9df8db0cc4376515785dee3123bd56dc2550a4c916b874e688176a"} Oct 30 04:17:22.876695 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.876590 4793 generic.go:356] "Generic (PLEG): container finished" podID="ab80769a-d645-4f51-bc10-f22ea546f798" containerID="c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60" exitCode=0 Oct 30 04:17:22.885503 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.881765 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" Oct 30 04:17:22.885503 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.883543 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" event={"ID":"ab80769a-d645-4f51-bc10-f22ea546f798","Type":"ContainerDied","Data":"c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60"} Oct 30 04:17:22.885503 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.883716 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx" event={"ID":"ab80769a-d645-4f51-bc10-f22ea546f798","Type":"ContainerDied","Data":"f3ad11a2aef58c81cadde2306aef17305ca52517aa1c71e76c6edcffbe1890ac"} Oct 30 04:17:22.895080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.895062 4793 scope.go:122] "RemoveContainer" containerID="51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb" Oct 30 04:17:22.895356 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:22.895329 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb\": container with ID starting with 51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb not found: ID does not exist" containerID="51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb" Oct 30 04:17:22.895491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.895352 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb"} err="failed to get container status \"51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb\": rpc error: code = NotFound desc = could not find container \"51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb\": container with ID starting with 51236af9485c8ae2af634308ce438a60c148e2ec0ec9b95c58f055ff823756bb not found: ID does not exist" Oct 30 04:17:22.895491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.895369 4793 scope.go:122] "RemoveContainer" containerID="c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60" Oct 30 04:17:22.910429 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.910396 4793 scope.go:122] "RemoveContainer" containerID="c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60" Oct 30 04:17:22.910783 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:22.910760 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60\": container with ID starting with c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60 not found: ID does not exist" containerID="c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60" Oct 30 04:17:22.910872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.910792 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60"} err="failed to get container status \"c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60\": rpc error: code = NotFound desc = could not find container \"c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60\": container with ID starting with c7144897e0e14612e08cacb7aeaf228dd2dab73357d2bb41d90cb1436fb70c60 not found: ID does not exist" Oct 30 04:17:22.951988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.951900 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-envoy\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.951988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.951951 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da898aa-742b-41a7-9c66-ab97460daf99-istio-podinfo\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.951988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.951978 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-istio-token\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.952284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.952014 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw957\" (UniqueName: \"kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-kube-api-access-hw957\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.952284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.952047 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-data\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.952284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.952074 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da898aa-742b-41a7-9c66-ab97460daf99-istiod-ca-cert\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.952284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.952165 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-certs\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.952284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.952195 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-credential-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.952284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.952236 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.952657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.952633 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.952724 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.952686 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-certs\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.952934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.952912 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da898aa-742b-41a7-9c66-ab97460daf99-istiod-ca-cert\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.953023 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.952914 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-credential-socket\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.953023 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.952976 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-data\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.967443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.957117 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da898aa-742b-41a7-9c66-ab97460daf99-istio-podinfo\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.972918 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.972841 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:22.983692 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.983657 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-envoy\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.985369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.985344 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-istio-token\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:22.987296 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:22.987272 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw957\" (UniqueName: \"kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-kube-api-access-hw957\") pod \"e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:23.016936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.016908 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx"] Oct 30 04:17:23.019279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.019259 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-de5733f9-744cdf6475-c6fkx"] Oct 30 04:17:23.052361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.052332 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt"] Oct 30 04:17:23.087714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.087681 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-de5733f9-55fc6fc67d-fcfmt"] Oct 30 04:17:23.134249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.134219 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:23.192270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.192235 4793 patch_prober.go:29] interesting pod/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 04:17:23.192464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.192285 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 04:17:23.193008 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.192974 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:23.488698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.488665 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r"] Oct 30 04:17:23.490863 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:23.490801 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da898aa_742b_41a7_9c66_ab97460daf99.slice/crio-eb18901bdbd1932d03e1328e979f4b23b8c4605e7af7c21d4b77a2b49283bcdd WatchSource:0}: Error finding container eb18901bdbd1932d03e1328e979f4b23b8c4605e7af7c21d4b77a2b49283bcdd: Status 404 returned error can't find the container with id eb18901bdbd1932d03e1328e979f4b23b8c4605e7af7c21d4b77a2b49283bcdd Oct 30 04:17:23.493937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.493900 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:23.494038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.493984 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:23.494038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.494027 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:23.500799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.498684 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:23.522691 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.522671 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz"] Oct 30 04:17:23.537642 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.537615 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4"] Oct 30 04:17:23.708468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.705047 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:23.900494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.899433 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" event={"ID":"2da898aa-742b-41a7-9c66-ab97460daf99","Type":"ContainerStarted","Data":"60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9"} Oct 30 04:17:23.900494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.899507 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" event={"ID":"2da898aa-742b-41a7-9c66-ab97460daf99","Type":"ContainerStarted","Data":"eb18901bdbd1932d03e1328e979f4b23b8c4605e7af7c21d4b77a2b49283bcdd"} Oct 30 04:17:23.917363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.917336 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:23.946706 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:23.946646 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" podStartSLOduration=1.946631912 podStartE2EDuration="1.946631912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:23.944992081 +0000 UTC m=+1167.230866856" watchObservedRunningTime="2026-10-30 04:17:23.946631912 +0000 UTC m=+1167.232506686" Oct 30 04:17:24.134550 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.134524 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:24.140024 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.140002 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:24.140261 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.140084 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:24.181326 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.181293 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:24.190881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.190858 4793 patch_prober.go:29] interesting pod/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Oct 30 04:17:24.191018 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.190905 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Oct 30 04:17:24.205353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.205333 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-openshift-ca\") pod \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\" (UID: \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\") " Oct 30 04:17:24.205513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.205477 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-xks-ca\") pod \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\" (UID: \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\") " Oct 30 04:17:24.205586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.205536 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa5f3fc-17e8-42ae-a04a-7e09c1894407-kube-api-access-tsvnd\" (UniqueName: \"kubernetes.io/projected/daa5f3fc-17e8-42ae-a04a-7e09c1894407-kube-api-access-tsvnd\") pod \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\" (UID: \"daa5f3fc-17e8-42ae-a04a-7e09c1894407\") " Oct 30 04:17:24.205749 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.205727 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-openshift-ca" pod "daa5f3fc-17e8-42ae-a04a-7e09c1894407" (UID: "daa5f3fc-17e8-42ae-a04a-7e09c1894407"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:24.205925 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.205904 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:24.208517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.208479 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-xks-ca" pod "daa5f3fc-17e8-42ae-a04a-7e09c1894407" (UID: "daa5f3fc-17e8-42ae-a04a-7e09c1894407"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:17:24.209019 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.208995 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa5f3fc-17e8-42ae-a04a-7e09c1894407-kube-api-access-tsvnd" pod "daa5f3fc-17e8-42ae-a04a-7e09c1894407" (UID: "daa5f3fc-17e8-42ae-a04a-7e09c1894407"). InnerVolumeSpecName "kube-api-access-tsvnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:24.327914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.314064 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsvnd\" (UniqueName: \"kubernetes.io/projected/daa5f3fc-17e8-42ae-a04a-7e09c1894407-kube-api-access-tsvnd\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:24.327914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.327848 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa5f3fc-17e8-42ae-a04a-7e09c1894407-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:24.394592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.394404 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:24.496145 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.496059 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:17:24.847404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.847311 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8a8f22-0902-4338-8991-7eaeeb963120" path="/var/lib/kubelet/pods/9b8a8f22-0902-4338-8991-7eaeeb963120/volumes" Oct 30 04:17:24.847847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.847819 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab80769a-d645-4f51-bc10-f22ea546f798" path="/var/lib/kubelet/pods/ab80769a-d645-4f51-bc10-f22ea546f798/volumes" Oct 30 04:17:24.916393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.916356 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" Oct 30 04:17:24.917070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.917040 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s" event={"ID":"daa5f3fc-17e8-42ae-a04a-7e09c1894407","Type":"ContainerDied","Data":"2a0c73ae9a9758d3d55808e1b818dd1834d59f44f3ca5a6c9b1ed34de3c82073"} Oct 30 04:17:24.917314 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.917295 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a0c73ae9a9758d3d55808e1b818dd1834d59f44f3ca5a6c9b1ed34de3c82073" Oct 30 04:17:24.919471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.919448 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:24.919760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:24.919742 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:25.194864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:25.194836 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:17:25.195936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:25.195916 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:17:25.239028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:25.238988 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:25.262569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:25.262531 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf"] Oct 30 04:17:25.264519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:25.264497 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad07bf73-7dxrf"] Oct 30 04:17:25.452022 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:25.451949 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:25.651306 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:25.651263 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:25.928208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:25.928180 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl"] Oct 30 04:17:25.928692 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:25.928353 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" containerID="cri-o://6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585" gracePeriod=30 Oct 30 04:17:26.068223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:26.066743 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:26.270548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:26.270461 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:26.485900 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:26.485867 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:26.845158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:26.845122 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28002b58-868c-4bcb-9e20-d4e2b4ef501a" path="/var/lib/kubelet/pods/28002b58-868c-4bcb-9e20-d4e2b4ef501a/volumes" Oct 30 04:17:26.846297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:26.846271 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:27.324633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:27.324595 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:27.594227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:27.594137 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:27.914744 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:27.914652 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:28.012712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.012675 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5"] Oct 30 04:17:28.013338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.013316 4793 cpu_manager.go:414] "containerMap: removing container" podUID="daa5f3fc-17e8-42ae-a04a-7e09c1894407" containerName="revoke-keys" Oct 30 04:17:28.013338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.013338 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa5f3fc-17e8-42ae-a04a-7e09c1894407" containerName="revoke-keys" Oct 30 04:17:28.013550 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.013502 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa5f3fc-17e8-42ae-a04a-7e09c1894407" containerName="revoke-keys" Oct 30 04:17:28.016934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.016914 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" Oct 30 04:17:28.034031 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.034008 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-b4708a-dockercfg-z6qp2\"" Oct 30 04:17:28.035905 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.035405 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5"] Oct 30 04:17:28.070446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.059973 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b4708a\"" Oct 30 04:17:28.124857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.124749 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:28.178922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.178896 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2xz6\" (UniqueName: \"kubernetes.io/projected/824a3b82-1a31-4f68-8d37-4fa51299579b-kube-api-access-z2xz6\") pod \"payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5\" (UID: \"824a3b82-1a31-4f68-8d37-4fa51299579b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" Oct 30 04:17:28.179091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.178935 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824a3b82-1a31-4f68-8d37-4fa51299579b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5\" (UID: \"824a3b82-1a31-4f68-8d37-4fa51299579b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" Oct 30 04:17:28.280343 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.280304 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2xz6\" (UniqueName: \"kubernetes.io/projected/824a3b82-1a31-4f68-8d37-4fa51299579b-kube-api-access-z2xz6\") pod \"payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5\" (UID: \"824a3b82-1a31-4f68-8d37-4fa51299579b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" Oct 30 04:17:28.280591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.280350 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824a3b82-1a31-4f68-8d37-4fa51299579b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5\" (UID: \"824a3b82-1a31-4f68-8d37-4fa51299579b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" Oct 30 04:17:28.281131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.281103 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824a3b82-1a31-4f68-8d37-4fa51299579b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5\" (UID: \"824a3b82-1a31-4f68-8d37-4fa51299579b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" Oct 30 04:17:28.318483 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.318409 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2xz6\" (UniqueName: \"kubernetes.io/projected/824a3b82-1a31-4f68-8d37-4fa51299579b-kube-api-access-z2xz6\") pod \"payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5\" (UID: \"824a3b82-1a31-4f68-8d37-4fa51299579b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" Oct 30 04:17:28.326050 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.326018 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:28.328907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.328884 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" Oct 30 04:17:28.523583 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.523472 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" podUID="d7100422-dab2-4920-80ad-fd6b7fef342a" containerName="payload-processing" containerID="cri-o://4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9" gracePeriod=25 Oct 30 04:17:28.537946 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.537908 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" podUID="ea3fa6fc-163a-483b-91b8-bce724c5254a" containerName="payload-pre-processing" containerID="cri-o://2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762" gracePeriod=25 Oct 30 04:17:28.626116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.626091 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5"] Oct 30 04:17:28.654741 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:28.654016 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824a3b82_1a31_4f68_8d37_4fa51299579b.slice/crio-22e32948feac31eb3443cb9587559f4202aef0737916737e0bb44fc8c9e6fe04 WatchSource:0}: Error finding container 22e32948feac31eb3443cb9587559f4202aef0737916737e0bb44fc8c9e6fe04: Status 404 returned error can't find the container with id 22e32948feac31eb3443cb9587559f4202aef0737916737e0bb44fc8c9e6fe04 Oct 30 04:17:28.782316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.782286 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:28.789161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.789135 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" Oct 30 04:17:28.843431 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.843388 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" Oct 30 04:17:28.887802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.887774 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7100422-dab2-4920-80ad-fd6b7fef342a-kube-api-access-tmxl4\" (UniqueName: \"kubernetes.io/projected/d7100422-dab2-4920-80ad-fd6b7fef342a-kube-api-access-tmxl4\") pod \"d7100422-dab2-4920-80ad-fd6b7fef342a\" (UID: \"d7100422-dab2-4920-80ad-fd6b7fef342a\") " Oct 30 04:17:28.887956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.887829 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7100422-dab2-4920-80ad-fd6b7fef342a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7100422-dab2-4920-80ad-fd6b7fef342a-plugins-config-volume\") pod \"d7100422-dab2-4920-80ad-fd6b7fef342a\" (UID: \"d7100422-dab2-4920-80ad-fd6b7fef342a\") " Oct 30 04:17:28.888347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.888323 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7100422-dab2-4920-80ad-fd6b7fef342a-plugins-config-volume" pod "d7100422-dab2-4920-80ad-fd6b7fef342a" (UID: "d7100422-dab2-4920-80ad-fd6b7fef342a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:28.890262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.890235 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7100422-dab2-4920-80ad-fd6b7fef342a-kube-api-access-tmxl4" pod "d7100422-dab2-4920-80ad-fd6b7fef342a" (UID: "d7100422-dab2-4920-80ad-fd6b7fef342a"). InnerVolumeSpecName "kube-api-access-tmxl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:28.943726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.943693 4793 generic.go:356] "Generic (PLEG): container finished" podID="d7100422-dab2-4920-80ad-fd6b7fef342a" containerID="4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9" exitCode=0 Oct 30 04:17:28.943847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.943758 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" Oct 30 04:17:28.943847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.943782 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" event={"ID":"d7100422-dab2-4920-80ad-fd6b7fef342a","Type":"ContainerDied","Data":"4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9"} Oct 30 04:17:28.943847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.943838 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz" event={"ID":"d7100422-dab2-4920-80ad-fd6b7fef342a","Type":"ContainerDied","Data":"6a18772c51f20671866087180e9397b3141c5396fdbb373ea7e957bda11da422"} Oct 30 04:17:28.943995 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.943870 4793 scope.go:122] "RemoveContainer" containerID="4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9" Oct 30 04:17:28.945500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.945475 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" event={"ID":"824a3b82-1a31-4f68-8d37-4fa51299579b","Type":"ContainerStarted","Data":"ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c"} Oct 30 04:17:28.945605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.945522 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" event={"ID":"824a3b82-1a31-4f68-8d37-4fa51299579b","Type":"ContainerStarted","Data":"22e32948feac31eb3443cb9587559f4202aef0737916737e0bb44fc8c9e6fe04"} Oct 30 04:17:28.945664 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.945645 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" Oct 30 04:17:28.947298 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.947275 4793 generic.go:356] "Generic (PLEG): container finished" podID="ea3fa6fc-163a-483b-91b8-bce724c5254a" containerID="2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762" exitCode=0 Oct 30 04:17:28.947404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.947318 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" Oct 30 04:17:28.947404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.947333 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" event={"ID":"ea3fa6fc-163a-483b-91b8-bce724c5254a","Type":"ContainerDied","Data":"2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762"} Oct 30 04:17:28.947404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.947376 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4" event={"ID":"ea3fa6fc-163a-483b-91b8-bce724c5254a","Type":"ContainerDied","Data":"21f2d0516898dfb69a938c40d74385b4fe3fdb3ee41605205555bf2a726f8840"} Oct 30 04:17:28.954822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.954798 4793 scope.go:122] "RemoveContainer" containerID="4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9" Oct 30 04:17:28.955112 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:28.955086 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9\": container with ID starting with 4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9 not found: ID does not exist" containerID="4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9" Oct 30 04:17:28.955186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.955120 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9"} err="failed to get container status \"4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9\": rpc error: code = NotFound desc = could not find container \"4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9\": container with ID starting with 4449ad31150494f05e5537ecf5990004ef6d5d7ec8670cdbb40973372ab56cc9 not found: ID does not exist" Oct 30 04:17:28.955186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.955141 4793 scope.go:122] "RemoveContainer" containerID="2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762" Oct 30 04:17:28.964830 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.964813 4793 scope.go:122] "RemoveContainer" containerID="2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762" Oct 30 04:17:28.965089 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:28.965069 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762\": container with ID starting with 2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762 not found: ID does not exist" containerID="2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762" Oct 30 04:17:28.965165 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.965094 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762"} err="failed to get container status \"2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762\": rpc error: code = NotFound desc = could not find container \"2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762\": container with ID starting with 2c638eacd9d2f4710c4bed939ad461950ba9ee366378a0674d9d287305932762 not found: ID does not exist" Oct 30 04:17:28.988633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.988606 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea3fa6fc-163a-483b-91b8-bce724c5254a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3fa6fc-163a-483b-91b8-bce724c5254a-plugins-config-volume\") pod \"ea3fa6fc-163a-483b-91b8-bce724c5254a\" (UID: \"ea3fa6fc-163a-483b-91b8-bce724c5254a\") " Oct 30 04:17:28.988748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.988730 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3fa6fc-163a-483b-91b8-bce724c5254a-kube-api-access-xrhcx\" (UniqueName: \"kubernetes.io/projected/ea3fa6fc-163a-483b-91b8-bce724c5254a-kube-api-access-xrhcx\") pod \"ea3fa6fc-163a-483b-91b8-bce724c5254a\" (UID: \"ea3fa6fc-163a-483b-91b8-bce724c5254a\") " Oct 30 04:17:28.988975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.988956 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3fa6fc-163a-483b-91b8-bce724c5254a-plugins-config-volume" pod "ea3fa6fc-163a-483b-91b8-bce724c5254a" (UID: "ea3fa6fc-163a-483b-91b8-bce724c5254a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:28.989102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.989081 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7100422-dab2-4920-80ad-fd6b7fef342a-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:28.989156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.989105 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3fa6fc-163a-483b-91b8-bce724c5254a-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:28.989156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.989121 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmxl4\" (UniqueName: \"kubernetes.io/projected/d7100422-dab2-4920-80ad-fd6b7fef342a-kube-api-access-tmxl4\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:28.990632 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:28.990609 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3fa6fc-163a-483b-91b8-bce724c5254a-kube-api-access-xrhcx" pod "ea3fa6fc-163a-483b-91b8-bce724c5254a" (UID: "ea3fa6fc-163a-483b-91b8-bce724c5254a"). InnerVolumeSpecName "kube-api-access-xrhcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:29.011911 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:29.011887 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz"] Oct 30 04:17:29.014362 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:29.014340 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ad07bf73-574bf7494b-6dksz"] Oct 30 04:17:29.053861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:29.053781 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" podStartSLOduration=2.053769064 podStartE2EDuration="2.053769064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:29.051650527 +0000 UTC m=+1172.337525308" watchObservedRunningTime="2026-10-30 04:17:29.053769064 +0000 UTC m=+1172.339643838" Oct 30 04:17:29.090306 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:29.090276 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrhcx\" (UniqueName: \"kubernetes.io/projected/ea3fa6fc-163a-483b-91b8-bce724c5254a-kube-api-access-xrhcx\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:29.298942 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:29.298880 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4"] Oct 30 04:17:29.303108 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:29.303083 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ad07bf73-6cc664945f-2cpp4"] Oct 30 04:17:30.110788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.110752 4793 patch_prober.go:29] interesting pod/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:17:30.111166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.110851 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:17:30.371682 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.371593 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:30.405739 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.405708 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58"] Oct 30 04:17:30.406154 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.406140 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ea3fa6fc-163a-483b-91b8-bce724c5254a" containerName="payload-pre-processing" Oct 30 04:17:30.406197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.406157 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3fa6fc-163a-483b-91b8-bce724c5254a" containerName="payload-pre-processing" Oct 30 04:17:30.406197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.406173 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d7100422-dab2-4920-80ad-fd6b7fef342a" containerName="payload-processing" Oct 30 04:17:30.406197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.406178 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7100422-dab2-4920-80ad-fd6b7fef342a" containerName="payload-processing" Oct 30 04:17:30.406291 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.406267 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3fa6fc-163a-483b-91b8-bce724c5254a" containerName="payload-pre-processing" Oct 30 04:17:30.406291 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.406275 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7100422-dab2-4920-80ad-fd6b7fef342a" containerName="payload-processing" Oct 30 04:17:30.409508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.409488 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" Oct 30 04:17:30.434544 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.434523 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58"] Oct 30 04:17:30.460020 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.459987 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7"] Oct 30 04:17:30.462827 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.462805 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" Oct 30 04:17:30.463398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.463371 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-a2f984\"" Oct 30 04:17:30.480113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.480091 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-a2f984-dockercfg-4g5h2\"" Oct 30 04:17:30.482437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.482402 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7"] Oct 30 04:17:30.502913 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.502883 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2mm6\" (UniqueName: \"kubernetes.io/projected/44cb423f-1206-4b8d-ad40-f5614e1da7fc-kube-api-access-h2mm6\") pod \"payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58\" (UID: \"44cb423f-1206-4b8d-ad40-f5614e1da7fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" Oct 30 04:17:30.503017 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.502996 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cb423f-1206-4b8d-ad40-f5614e1da7fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58\" (UID: \"44cb423f-1206-4b8d-ad40-f5614e1da7fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" Oct 30 04:17:30.503170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.503151 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7\" (UID: \"3c27f602-eb4b-4a29-8b95-ac7b9d736e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" Oct 30 04:17:30.503276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.503258 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf8nj\" (UniqueName: \"kubernetes.io/projected/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-kube-api-access-gf8nj\") pod \"payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7\" (UID: \"3c27f602-eb4b-4a29-8b95-ac7b9d736e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" Oct 30 04:17:30.567510 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.566086 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:30.603838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.603805 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf8nj\" (UniqueName: \"kubernetes.io/projected/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-kube-api-access-gf8nj\") pod \"payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7\" (UID: \"3c27f602-eb4b-4a29-8b95-ac7b9d736e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" Oct 30 04:17:30.603980 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.603852 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mm6\" (UniqueName: \"kubernetes.io/projected/44cb423f-1206-4b8d-ad40-f5614e1da7fc-kube-api-access-h2mm6\") pod \"payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58\" (UID: \"44cb423f-1206-4b8d-ad40-f5614e1da7fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" Oct 30 04:17:30.603980 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.603896 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cb423f-1206-4b8d-ad40-f5614e1da7fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58\" (UID: \"44cb423f-1206-4b8d-ad40-f5614e1da7fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" Oct 30 04:17:30.604100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.603987 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7\" (UID: \"3c27f602-eb4b-4a29-8b95-ac7b9d736e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" Oct 30 04:17:30.604707 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.604683 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cb423f-1206-4b8d-ad40-f5614e1da7fc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58\" (UID: \"44cb423f-1206-4b8d-ad40-f5614e1da7fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" Oct 30 04:17:30.604814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.604693 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7\" (UID: \"3c27f602-eb4b-4a29-8b95-ac7b9d736e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" Oct 30 04:17:30.627451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.627375 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mm6\" (UniqueName: \"kubernetes.io/projected/44cb423f-1206-4b8d-ad40-f5614e1da7fc-kube-api-access-h2mm6\") pod \"payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58\" (UID: \"44cb423f-1206-4b8d-ad40-f5614e1da7fc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" Oct 30 04:17:30.627451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.627382 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf8nj\" (UniqueName: \"kubernetes.io/projected/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-kube-api-access-gf8nj\") pod \"payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7\" (UID: \"3c27f602-eb4b-4a29-8b95-ac7b9d736e88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" Oct 30 04:17:30.721703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.721669 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" Oct 30 04:17:30.769157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.769128 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:30.775144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.775116 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" Oct 30 04:17:30.847853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.847820 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7100422-dab2-4920-80ad-fd6b7fef342a" path="/var/lib/kubelet/pods/d7100422-dab2-4920-80ad-fd6b7fef342a/volumes" Oct 30 04:17:30.848304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:30.848285 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3fa6fc-163a-483b-91b8-bce724c5254a" path="/var/lib/kubelet/pods/ea3fa6fc-163a-483b-91b8-bce724c5254a/volumes" Oct 30 04:17:31.023994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.023966 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58"] Oct 30 04:17:31.024798 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:31.024769 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44cb423f_1206_4b8d_ad40_f5614e1da7fc.slice/crio-4fb16e9d0cd7111aa4665ffbe33b9084b370b98cc9cbcd8b1af7fbfebe486030 WatchSource:0}: Error finding container 4fb16e9d0cd7111aa4665ffbe33b9084b370b98cc9cbcd8b1af7fbfebe486030: Status 404 returned error can't find the container with id 4fb16e9d0cd7111aa4665ffbe33b9084b370b98cc9cbcd8b1af7fbfebe486030 Oct 30 04:17:31.102942 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:31.102435 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c27f602_eb4b_4a29_8b95_ac7b9d736e88.slice/crio-116f4f7c72fe26f24f3c8322c272868b6a6eeebc02cfb2d1634cce5f65d4f9bd WatchSource:0}: Error finding container 116f4f7c72fe26f24f3c8322c272868b6a6eeebc02cfb2d1634cce5f65d4f9bd: Status 404 returned error can't find the container with id 116f4f7c72fe26f24f3c8322c272868b6a6eeebc02cfb2d1634cce5f65d4f9bd Oct 30 04:17:31.103106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.103042 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7"] Oct 30 04:17:31.199713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.199689 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:17:31.213001 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.212978 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-data\") pod \"d8ab6828-4959-4412-9fdb-50237628cd8d\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " Oct 30 04:17:31.213123 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213029 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-socket\") pod \"d8ab6828-4959-4412-9fdb-50237628cd8d\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " Oct 30 04:17:31.213123 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213088 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-podinfo\") pod \"d8ab6828-4959-4412-9fdb-50237628cd8d\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " Oct 30 04:17:31.213302 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213180 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-kube-api-access-7rkq4\" (UniqueName: \"kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-kube-api-access-7rkq4\") pod \"d8ab6828-4959-4412-9fdb-50237628cd8d\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " Oct 30 04:17:31.213302 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213224 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ab6828-4959-4412-9fdb-50237628cd8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ab6828-4959-4412-9fdb-50237628cd8d-istiod-ca-cert\") pod \"d8ab6828-4959-4412-9fdb-50237628cd8d\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " Oct 30 04:17:31.213302 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213261 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-envoy\") pod \"d8ab6828-4959-4412-9fdb-50237628cd8d\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " Oct 30 04:17:31.213302 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213277 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-data" pod "d8ab6828-4959-4412-9fdb-50237628cd8d" (UID: "d8ab6828-4959-4412-9fdb-50237628cd8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:31.213531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213286 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-socket" pod "d8ab6828-4959-4412-9fdb-50237628cd8d" (UID: "d8ab6828-4959-4412-9fdb-50237628cd8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:31.213531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213316 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-credential-socket\") pod \"d8ab6828-4959-4412-9fdb-50237628cd8d\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " Oct 30 04:17:31.213531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213356 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-certs\") pod \"d8ab6828-4959-4412-9fdb-50237628cd8d\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " Oct 30 04:17:31.213531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213432 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-token\" (UniqueName: \"kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-token\") pod \"d8ab6828-4959-4412-9fdb-50237628cd8d\" (UID: \"d8ab6828-4959-4412-9fdb-50237628cd8d\") " Oct 30 04:17:31.213761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213631 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ab6828-4959-4412-9fdb-50237628cd8d-istiod-ca-cert" pod "d8ab6828-4959-4412-9fdb-50237628cd8d" (UID: "d8ab6828-4959-4412-9fdb-50237628cd8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:31.213761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213703 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-certs" pod "d8ab6828-4959-4412-9fdb-50237628cd8d" (UID: "d8ab6828-4959-4412-9fdb-50237628cd8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:31.213834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213771 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-credential-socket" pod "d8ab6828-4959-4412-9fdb-50237628cd8d" (UID: "d8ab6828-4959-4412-9fdb-50237628cd8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:31.213882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213831 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8ab6828-4959-4412-9fdb-50237628cd8d-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:31.213882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213851 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:31.213882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213865 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:31.213882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.213878 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:31.215504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.215476 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-podinfo" pod "d8ab6828-4959-4412-9fdb-50237628cd8d" (UID: "d8ab6828-4959-4412-9fdb-50237628cd8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:17:31.215758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.215738 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-kube-api-access-7rkq4" pod "d8ab6828-4959-4412-9fdb-50237628cd8d" (UID: "d8ab6828-4959-4412-9fdb-50237628cd8d"). InnerVolumeSpecName "kube-api-access-7rkq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:31.215876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.215858 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-token" pod "d8ab6828-4959-4412-9fdb-50237628cd8d" (UID: "d8ab6828-4959-4412-9fdb-50237628cd8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:31.215915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.215861 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-envoy" pod "d8ab6828-4959-4412-9fdb-50237628cd8d" (UID: "d8ab6828-4959-4412-9fdb-50237628cd8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:31.270833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.270804 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:31.315216 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.315183 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:31.315216 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.315208 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:31.315216 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.315218 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:31.315538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.315227 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rkq4\" (UniqueName: \"kubernetes.io/projected/d8ab6828-4959-4412-9fdb-50237628cd8d-kube-api-access-7rkq4\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:31.315538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.315236 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8ab6828-4959-4412-9fdb-50237628cd8d-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:31.474129 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.474046 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:31.982287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.982249 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" event={"ID":"3c27f602-eb4b-4a29-8b95-ac7b9d736e88","Type":"ContainerStarted","Data":"74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c"} Oct 30 04:17:31.982517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.982305 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" event={"ID":"3c27f602-eb4b-4a29-8b95-ac7b9d736e88","Type":"ContainerStarted","Data":"116f4f7c72fe26f24f3c8322c272868b6a6eeebc02cfb2d1634cce5f65d4f9bd"} Oct 30 04:17:31.982517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.982373 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" Oct 30 04:17:31.984150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.984123 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" event={"ID":"44cb423f-1206-4b8d-ad40-f5614e1da7fc","Type":"ContainerStarted","Data":"37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb"} Oct 30 04:17:31.984272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.984164 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" event={"ID":"44cb423f-1206-4b8d-ad40-f5614e1da7fc","Type":"ContainerStarted","Data":"4fb16e9d0cd7111aa4665ffbe33b9084b370b98cc9cbcd8b1af7fbfebe486030"} Oct 30 04:17:31.984272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.984235 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" Oct 30 04:17:31.985834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.985810 4793 generic.go:356] "Generic (PLEG): container finished" podID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerID="6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585" exitCode=0 Oct 30 04:17:31.985936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.985857 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" Oct 30 04:17:31.985936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.985891 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" event={"ID":"d8ab6828-4959-4412-9fdb-50237628cd8d","Type":"ContainerDied","Data":"6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585"} Oct 30 04:17:31.986064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.985940 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl" event={"ID":"d8ab6828-4959-4412-9fdb-50237628cd8d","Type":"ContainerDied","Data":"91eaf2dfabcf355eca7c238f11e0d73e8e16613b7215baa51bec5da451730ac6"} Oct 30 04:17:31.986064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.985970 4793 scope.go:122] "RemoveContainer" containerID="6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585" Oct 30 04:17:31.996902 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:31.996823 4793 scope.go:122] "RemoveContainer" containerID="4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5" Oct 30 04:17:32.005577 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:32.005554 4793 scope.go:122] "RemoveContainer" containerID="6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585" Oct 30 04:17:32.005840 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:32.005822 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585\": container with ID starting with 6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585 not found: ID does not exist" containerID="6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585" Oct 30 04:17:32.005896 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:32.005848 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585"} err="failed to get container status \"6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585\": rpc error: code = NotFound desc = could not find container \"6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585\": container with ID starting with 6b30d4a6b1d5ab60145b935dfd778b4ae5b4fd0f23019f2ca84513c9cf28b585 not found: ID does not exist" Oct 30 04:17:32.005896 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:32.005864 4793 scope.go:122] "RemoveContainer" containerID="4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5" Oct 30 04:17:32.006086 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:32.006072 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5\": container with ID starting with 4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5 not found: ID does not exist" containerID="4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5" Oct 30 04:17:32.006125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:32.006089 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5"} err="failed to get container status \"4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5\": rpc error: code = NotFound desc = could not find container \"4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5\": container with ID starting with 4e6f0fcaec09ae255bd68319beeb5b1fc364a1a258c2b4edd7878a89e3f805c5 not found: ID does not exist" Oct 30 04:17:32.026344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:32.026304 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" podStartSLOduration=2.026289103 podStartE2EDuration="2.026289103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:32.026151139 +0000 UTC m=+1175.312025915" watchObservedRunningTime="2026-10-30 04:17:32.026289103 +0000 UTC m=+1175.312163879" Oct 30 04:17:32.067970 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:32.067934 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" podStartSLOduration=2.067922601 podStartE2EDuration="2.067922601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:32.066583103 +0000 UTC m=+1175.352457882" watchObservedRunningTime="2026-10-30 04:17:32.067922601 +0000 UTC m=+1175.353797375" Oct 30 04:17:32.104182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:32.104156 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl"] Oct 30 04:17:32.116123 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:32.116102 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ad07bf73-openshift-default-6c8fddcf7c-bspbl"] Oct 30 04:17:32.341661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:32.341573 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:32.846413 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:32.846381 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" path="/var/lib/kubelet/pods/d8ab6828-4959-4412-9fdb-50237628cd8d/volumes" Oct 30 04:17:32.920854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:32.920817 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n"] Oct 30 04:17:32.951064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:32.951034 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj"] Oct 30 04:17:33.151545 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:33.151455 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:33.350535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:33.350507 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:34.191235 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.191191 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:34.350791 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.350758 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q"] Oct 30 04:17:34.351235 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.351222 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" Oct 30 04:17:34.351275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.351236 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" Oct 30 04:17:34.351312 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.351305 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" Oct 30 04:17:34.351347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.351316 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" Oct 30 04:17:34.354268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.354249 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" Oct 30 04:17:34.369720 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.369699 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-36bbac-dockercfg-jhzrj\"" Oct 30 04:17:34.382232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.382213 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-36bbac\"" Oct 30 04:17:34.386536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.386513 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q"] Oct 30 04:17:34.397484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.397456 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:34.438919 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.438879 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg"] Oct 30 04:17:34.460201 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.453851 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v27lx\" (UniqueName: \"kubernetes.io/projected/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-kube-api-access-v27lx\") pod \"payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q\" (UID: \"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" Oct 30 04:17:34.460201 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.459815 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q\" (UID: \"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" Oct 30 04:17:34.461373 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.461341 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" Oct 30 04:17:34.461568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.461551 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ab6828-4959-4412-9fdb-50237628cd8d" containerName="istio-proxy" Oct 30 04:17:34.478563 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.478524 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg"] Oct 30 04:17:34.478703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.478622 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" Oct 30 04:17:34.561084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.561052 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v27lx\" (UniqueName: \"kubernetes.io/projected/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-kube-api-access-v27lx\") pod \"payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q\" (UID: \"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" Oct 30 04:17:34.561266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.561091 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s95ps\" (UniqueName: \"kubernetes.io/projected/f186039e-f254-4792-bea7-3607de3d4a7f-kube-api-access-s95ps\") pod \"payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg\" (UID: \"f186039e-f254-4792-bea7-3607de3d4a7f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" Oct 30 04:17:34.561266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.561113 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q\" (UID: \"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" Oct 30 04:17:34.561376 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.561278 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f186039e-f254-4792-bea7-3607de3d4a7f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg\" (UID: \"f186039e-f254-4792-bea7-3607de3d4a7f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" Oct 30 04:17:34.561860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.561838 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q\" (UID: \"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" Oct 30 04:17:34.584895 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.584864 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v27lx\" (UniqueName: \"kubernetes.io/projected/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-kube-api-access-v27lx\") pod \"payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q\" (UID: \"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" Oct 30 04:17:34.601785 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.601765 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:34.662793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.662771 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f186039e-f254-4792-bea7-3607de3d4a7f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg\" (UID: \"f186039e-f254-4792-bea7-3607de3d4a7f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" Oct 30 04:17:34.662898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.662821 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s95ps\" (UniqueName: \"kubernetes.io/projected/f186039e-f254-4792-bea7-3607de3d4a7f-kube-api-access-s95ps\") pod \"payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg\" (UID: \"f186039e-f254-4792-bea7-3607de3d4a7f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" Oct 30 04:17:34.663402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.663382 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f186039e-f254-4792-bea7-3607de3d4a7f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg\" (UID: \"f186039e-f254-4792-bea7-3607de3d4a7f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" Oct 30 04:17:34.666627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.666608 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" Oct 30 04:17:34.686234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.686208 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s95ps\" (UniqueName: \"kubernetes.io/projected/f186039e-f254-4792-bea7-3607de3d4a7f-kube-api-access-s95ps\") pod \"payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg\" (UID: \"f186039e-f254-4792-bea7-3607de3d4a7f\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" Oct 30 04:17:34.790802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.790772 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" Oct 30 04:17:34.966812 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:34.966785 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c14d0b_ca18_4835_94c4_aedc0c1ad8e7.slice/crio-c02f887a37129a07dc4a7d1113f6bd6aac485b350732e8099ecf1da95ae333d1 WatchSource:0}: Error finding container c02f887a37129a07dc4a7d1113f6bd6aac485b350732e8099ecf1da95ae333d1: Status 404 returned error can't find the container with id c02f887a37129a07dc4a7d1113f6bd6aac485b350732e8099ecf1da95ae333d1 Oct 30 04:17:34.967049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:34.967027 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q"] Oct 30 04:17:35.012966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:35.012941 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" event={"ID":"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7","Type":"ContainerStarted","Data":"c02f887a37129a07dc4a7d1113f6bd6aac485b350732e8099ecf1da95ae333d1"} Oct 30 04:17:35.091786 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:35.091757 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf186039e_f254_4792_bea7_3607de3d4a7f.slice/crio-50dcda9d96d1daebb8bd03ae72b0f59639271595dd403dd6fb92feeac8e0208a WatchSource:0}: Error finding container 50dcda9d96d1daebb8bd03ae72b0f59639271595dd403dd6fb92feeac8e0208a: Status 404 returned error can't find the container with id 50dcda9d96d1daebb8bd03ae72b0f59639271595dd403dd6fb92feeac8e0208a Oct 30 04:17:35.091880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:35.091808 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg"] Oct 30 04:17:35.699707 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:35.699665 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:36.020819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.020722 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" event={"ID":"f186039e-f254-4792-bea7-3607de3d4a7f","Type":"ContainerStarted","Data":"f989dc363e40c8c9888871988145afe8f7972bee92b8e0c080799992e007fa53"} Oct 30 04:17:36.020819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.020791 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" Oct 30 04:17:36.020819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.020815 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" event={"ID":"f186039e-f254-4792-bea7-3607de3d4a7f","Type":"ContainerStarted","Data":"50dcda9d96d1daebb8bd03ae72b0f59639271595dd403dd6fb92feeac8e0208a"} Oct 30 04:17:36.022465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.022406 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" event={"ID":"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7","Type":"ContainerStarted","Data":"ae3dee5aa118ee3dbcc4ccc2a67d9858a0d7825ed1482705d799877d15cfe1f3"} Oct 30 04:17:36.022568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.022493 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" Oct 30 04:17:36.064678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.064633 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" podStartSLOduration=2.064620319 podStartE2EDuration="2.064620319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:36.062124217 +0000 UTC m=+1179.347999018" watchObservedRunningTime="2026-10-30 04:17:36.064620319 +0000 UTC m=+1179.350495088" Oct 30 04:17:36.105439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.105367 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" podStartSLOduration=2.105353205 podStartE2EDuration="2.105353205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:36.103318758 +0000 UTC m=+1179.389193535" watchObservedRunningTime="2026-10-30 04:17:36.105353205 +0000 UTC m=+1179.391227979" Oct 30 04:17:36.189385 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.189344 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:36.223324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.223259 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s"] Oct 30 04:17:36.225590 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.225567 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-cb1a18be-4mj7s"] Oct 30 04:17:36.375581 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.375495 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h"] Oct 30 04:17:36.375750 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.375698 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" containerID="cri-o://da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8" gracePeriod=30 Oct 30 04:17:36.395214 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.395189 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:36.593878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.593844 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:36.847689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:36.847651 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa5f3fc-17e8-42ae-a04a-7e09c1894407" path="/var/lib/kubelet/pods/daa5f3fc-17e8-42ae-a04a-7e09c1894407/volumes" Oct 30 04:17:37.380313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.380251 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:37.731551 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.731514 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm"] Oct 30 04:17:37.736651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.736625 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.753400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.753349 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6796b1-openshift-default-dockercfg-h254p\"" Oct 30 04:17:37.768551 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.768527 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm"] Oct 30 04:17:37.798918 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.798894 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.799022 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.798979 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0886d884-9043-4114-ba66-829bf6f319b7-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.799071 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.799024 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0886d884-9043-4114-ba66-829bf6f319b7-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.799071 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.799059 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.799174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.799156 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.799220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.799208 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.799296 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.799284 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.799332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.799307 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.799363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.799347 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbj8h\" (UniqueName: \"kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-kube-api-access-vbj8h\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.899993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.899962 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.900464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.900002 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.900464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.900080 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.900464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.900113 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.900464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.900173 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbj8h\" (UniqueName: \"kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-kube-api-access-vbj8h\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.900464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.900221 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.900464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.900263 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0886d884-9043-4114-ba66-829bf6f319b7-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.900464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.900299 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0886d884-9043-4114-ba66-829bf6f319b7-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.900464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.900347 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.900877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.900745 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.900877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.900775 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.900877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.900815 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.901074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.901047 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.901331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.901309 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0886d884-9043-4114-ba66-829bf6f319b7-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.902475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.902452 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.902874 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.902846 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0886d884-9043-4114-ba66-829bf6f319b7-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.904348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.904322 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:37.922071 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.922040 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" podUID="103c67b2-cc79-46fa-a2f4-43afa4553edb" containerName="payload-processing" containerID="cri-o://c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c" gracePeriod=25 Oct 30 04:17:37.927733 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.927708 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.930331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.930304 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbj8h\" (UniqueName: \"kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-kube-api-access-vbj8h\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:37.954469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:37.951867 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" podUID="452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51" containerName="payload-pre-processing" containerID="cri-o://c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9" gracePeriod=25 Oct 30 04:17:38.052402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.052374 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:38.117993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.117936 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:38.278092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.278067 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" Oct 30 04:17:38.288888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.288865 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" Oct 30 04:17:38.304709 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.304686 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103c67b2-cc79-46fa-a2f4-43afa4553edb-kube-api-access-lnflz\" (UniqueName: \"kubernetes.io/projected/103c67b2-cc79-46fa-a2f4-43afa4553edb-kube-api-access-lnflz\") pod \"103c67b2-cc79-46fa-a2f4-43afa4553edb\" (UID: \"103c67b2-cc79-46fa-a2f4-43afa4553edb\") " Oct 30 04:17:38.304841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.304763 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103c67b2-cc79-46fa-a2f4-43afa4553edb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103c67b2-cc79-46fa-a2f4-43afa4553edb-plugins-config-volume\") pod \"103c67b2-cc79-46fa-a2f4-43afa4553edb\" (UID: \"103c67b2-cc79-46fa-a2f4-43afa4553edb\") " Oct 30 04:17:38.305258 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.305222 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103c67b2-cc79-46fa-a2f4-43afa4553edb-plugins-config-volume" pod "103c67b2-cc79-46fa-a2f4-43afa4553edb" (UID: "103c67b2-cc79-46fa-a2f4-43afa4553edb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:38.306993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.306971 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103c67b2-cc79-46fa-a2f4-43afa4553edb-kube-api-access-lnflz" pod "103c67b2-cc79-46fa-a2f4-43afa4553edb" (UID: "103c67b2-cc79-46fa-a2f4-43afa4553edb"). InnerVolumeSpecName "kube-api-access-lnflz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:38.332299 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.332262 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:38.379466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.378436 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm"] Oct 30 04:17:38.389547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.389500 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:38.389725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.389595 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:38.389725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.389639 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:38.409482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.405878 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-plugins-config-volume\") pod \"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51\" (UID: \"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51\") " Oct 30 04:17:38.409482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.405952 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-kube-api-access-dm7sl\" (UniqueName: \"kubernetes.io/projected/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-kube-api-access-dm7sl\") pod \"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51\" (UID: \"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51\") " Oct 30 04:17:38.409482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.406264 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-plugins-config-volume" pod "452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51" (UID: "452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:38.409482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.406292 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnflz\" (UniqueName: \"kubernetes.io/projected/103c67b2-cc79-46fa-a2f4-43afa4553edb-kube-api-access-lnflz\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:38.409482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.406310 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103c67b2-cc79-46fa-a2f4-43afa4553edb-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:38.410804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.409981 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-kube-api-access-dm7sl" pod "452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51" (UID: "452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51"). InnerVolumeSpecName "kube-api-access-dm7sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:38.508099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.508065 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:38.508099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.508102 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm7sl\" (UniqueName: \"kubernetes.io/projected/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51-kube-api-access-dm7sl\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:38.527582 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.527515 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:38.995938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:38.995896 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:39.040996 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.040920 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" event={"ID":"0886d884-9043-4114-ba66-829bf6f319b7","Type":"ContainerStarted","Data":"e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a"} Oct 30 04:17:39.041299 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.041257 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" event={"ID":"0886d884-9043-4114-ba66-829bf6f319b7","Type":"ContainerStarted","Data":"5b23928b7f1a65736c75eddf899ce0b8774cf9528a700751eeaeb027eb737186"} Oct 30 04:17:39.043185 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.043155 4793 generic.go:356] "Generic (PLEG): container finished" podID="103c67b2-cc79-46fa-a2f4-43afa4553edb" containerID="c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c" exitCode=0 Oct 30 04:17:39.043316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.043215 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" Oct 30 04:17:39.043316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.043300 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" event={"ID":"103c67b2-cc79-46fa-a2f4-43afa4553edb","Type":"ContainerDied","Data":"c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c"} Oct 30 04:17:39.043406 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.043337 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n" event={"ID":"103c67b2-cc79-46fa-a2f4-43afa4553edb","Type":"ContainerDied","Data":"0e35b52d88ca37e7fe3c601271c0f5a33d3634ecf515b4c6e84158a2ffea1c94"} Oct 30 04:17:39.043406 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.043368 4793 scope.go:122] "RemoveContainer" containerID="c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c" Oct 30 04:17:39.052673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.052650 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:39.075948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.075747 4793 generic.go:356] "Generic (PLEG): container finished" podID="452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51" containerID="c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9" exitCode=0 Oct 30 04:17:39.075948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.075808 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" Oct 30 04:17:39.075948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.075851 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" event={"ID":"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51","Type":"ContainerDied","Data":"c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9"} Oct 30 04:17:39.075948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.075897 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj" event={"ID":"452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51","Type":"ContainerDied","Data":"1844fde47effbc37d5c2ce4ea8c8672b27a5da5956f605f4150b2e60e48cdae1"} Oct 30 04:17:39.084743 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.084721 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:39.095927 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.095904 4793 scope.go:122] "RemoveContainer" containerID="c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c" Oct 30 04:17:39.096260 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:39.096241 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c\": container with ID starting with c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c not found: ID does not exist" containerID="c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c" Oct 30 04:17:39.096306 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.096270 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c"} err="failed to get container status \"c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c\": rpc error: code = NotFound desc = could not find container \"c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c\": container with ID starting with c1bd36f9ae26f830725d60cf866fc884843592ca6df1dab83435b39a76df470c not found: ID does not exist" Oct 30 04:17:39.096306 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.096288 4793 scope.go:122] "RemoveContainer" containerID="c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9" Oct 30 04:17:39.101594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.101547 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" podStartSLOduration=2.101531063 podStartE2EDuration="2.101531063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:39.101053663 +0000 UTC m=+1182.386928460" watchObservedRunningTime="2026-10-30 04:17:39.101531063 +0000 UTC m=+1182.387405839" Oct 30 04:17:39.105552 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.105533 4793 scope.go:122] "RemoveContainer" containerID="c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9" Oct 30 04:17:39.105833 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:39.105803 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9\": container with ID starting with c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9 not found: ID does not exist" containerID="c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9" Oct 30 04:17:39.105924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.105842 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9"} err="failed to get container status \"c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9\": rpc error: code = NotFound desc = could not find container \"c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9\": container with ID starting with c9a5c4311441475723e9dbe705adcfdd383b4071dfd921f25f24b1f314a5cca9 not found: ID does not exist" Oct 30 04:17:39.228678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.228612 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj"] Oct 30 04:17:39.232103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.232070 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-cb1a18be-94d4b77c5-9nhwj"] Oct 30 04:17:39.283174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.283132 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n"] Oct 30 04:17:39.285951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.285928 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-cb1a18be-dd97f446b-j2p8n"] Oct 30 04:17:39.528392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.528314 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:39.746202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.746160 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:39.976607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:39.976572 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" Oct 30 04:17:40.105813 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.105707 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:40.106847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.106818 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:17:40.198951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.197919 4793 patch_prober.go:29] interesting pod/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:17:40.198951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.198014 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:17:40.344069 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.343989 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:40.846083 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.846049 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103c67b2-cc79-46fa-a2f4-43afa4553edb" path="/var/lib/kubelet/pods/103c67b2-cc79-46fa-a2f4-43afa4553edb/volumes" Oct 30 04:17:40.846451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.846438 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51" path="/var/lib/kubelet/pods/452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51/volumes" Oct 30 04:17:40.937639 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.937604 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq"] Oct 30 04:17:40.938312 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.938285 4793 cpu_manager.go:414] "containerMap: removing container" podUID="103c67b2-cc79-46fa-a2f4-43afa4553edb" containerName="payload-processing" Oct 30 04:17:40.938312 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.938311 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103c67b2-cc79-46fa-a2f4-43afa4553edb" containerName="payload-processing" Oct 30 04:17:40.938545 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.938381 4793 cpu_manager.go:414] "containerMap: removing container" podUID="452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51" containerName="payload-pre-processing" Oct 30 04:17:40.938545 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.938391 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51" containerName="payload-pre-processing" Oct 30 04:17:40.938545 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.938508 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="452ddb1f-4bc8-4dcb-8dfc-2f0e6ff72f51" containerName="payload-pre-processing" Oct 30 04:17:40.938545 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.938524 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="103c67b2-cc79-46fa-a2f4-43afa4553edb" containerName="payload-processing" Oct 30 04:17:40.941884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.941861 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:40.996572 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.996542 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5d74b9\"/\"kube-root-ca.crt\"" Oct 30 04:17:40.996780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.996592 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5d74b9\"/\"simulator-w2-5d74b9-kserve-self-signed-certs\"" Oct 30 04:17:40.996780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.996660 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5d74b9\"/\"default-dockercfg-j7cqj\"" Oct 30 04:17:40.996780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.996695 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5d74b9\"/\"openshift-service-ca.crt\"" Oct 30 04:17:40.999379 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:40.999355 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq"] Oct 30 04:17:41.034090 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.034062 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-model-cache\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.034207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.034118 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-dshm\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.034277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.034223 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgzmn\" (UniqueName: \"kubernetes.io/projected/e70bca52-d177-4b80-b29b-6e0d64549ef0-kube-api-access-cgzmn\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.034277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.034261 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-home\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.034386 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.034366 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-tmp-dir\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.034457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.034389 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70bca52-d177-4b80-b29b-6e0d64549ef0-tls-certs\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.117889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.117788 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:41.134928 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.134898 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-dshm\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.135026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.134941 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgzmn\" (UniqueName: \"kubernetes.io/projected/e70bca52-d177-4b80-b29b-6e0d64549ef0-kube-api-access-cgzmn\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.135026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.134962 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-home\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.135117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.135025 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-tmp-dir\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.135117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.135044 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70bca52-d177-4b80-b29b-6e0d64549ef0-tls-certs\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.135227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.135125 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-model-cache\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.135411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.135380 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-home\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.135574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.135471 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-model-cache\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.135574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.135537 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-tmp-dir\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.137232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.137202 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-dshm\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.137506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.137488 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70bca52-d177-4b80-b29b-6e0d64549ef0-tls-certs\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.179247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.179220 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgzmn\" (UniqueName: \"kubernetes.io/projected/e70bca52-d177-4b80-b29b-6e0d64549ef0-kube-api-access-cgzmn\") pod \"simulator-w2-5d74b9-kserve-cb6745f6-f5rjq\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.231321 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:41.231275 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:17:41.284807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.284776 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:41.589316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.589285 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq"] Oct 30 04:17:41.589858 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:41.589828 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode70bca52_d177_4b80_b29b_6e0d64549ef0.slice/crio-d812c1e62139555401751cf52e21899b921bb8b058db863e07ca3e886f67dc3e WatchSource:0}: Error finding container d812c1e62139555401751cf52e21899b921bb8b058db863e07ca3e886f67dc3e: Status 404 returned error can't find the container with id d812c1e62139555401751cf52e21899b921bb8b058db863e07ca3e886f67dc3e Oct 30 04:17:41.701718 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.701688 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:41.923464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:41.923348 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:42.035741 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.035720 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:17:42.119784 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.119750 4793 generic.go:356] "Generic (PLEG): container finished" podID="ba448995-8974-4206-88a6-2c4f0259759f" containerID="da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8" exitCode=0 Oct 30 04:17:42.120204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.119806 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" Oct 30 04:17:42.120204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.119832 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" event={"ID":"ba448995-8974-4206-88a6-2c4f0259759f","Type":"ContainerDied","Data":"da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8"} Oct 30 04:17:42.120204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.119886 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h" event={"ID":"ba448995-8974-4206-88a6-2c4f0259759f","Type":"ContainerDied","Data":"ea069fa3c8fabbc3c142c3a9dc636bcc1e2ed10322e7e77542426cc8a50e8e8d"} Oct 30 04:17:42.120204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.119916 4793 scope.go:122] "RemoveContainer" containerID="da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8" Oct 30 04:17:42.121809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.121779 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" event={"ID":"e70bca52-d177-4b80-b29b-6e0d64549ef0","Type":"ContainerStarted","Data":"0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f"} Oct 30 04:17:42.122000 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.121820 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" event={"ID":"e70bca52-d177-4b80-b29b-6e0d64549ef0","Type":"ContainerStarted","Data":"d812c1e62139555401751cf52e21899b921bb8b058db863e07ca3e886f67dc3e"} Oct 30 04:17:42.130714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.130693 4793 scope.go:122] "RemoveContainer" containerID="88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939" Oct 30 04:17:42.141894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.141810 4793 scope.go:122] "RemoveContainer" containerID="da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8" Oct 30 04:17:42.142249 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:42.142226 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8\": container with ID starting with da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8 not found: ID does not exist" containerID="da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8" Oct 30 04:17:42.142347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.142262 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8"} err="failed to get container status \"da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8\": rpc error: code = NotFound desc = could not find container \"da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8\": container with ID starting with da0213c44c74ed99f622b8b40ca0341291dc787bc5e23387dc37d39f4d70b8c8 not found: ID does not exist" Oct 30 04:17:42.142347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.142287 4793 scope.go:122] "RemoveContainer" containerID="88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939" Oct 30 04:17:42.142778 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:17:42.142717 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939\": container with ID starting with 88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939 not found: ID does not exist" containerID="88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939" Oct 30 04:17:42.142778 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.142755 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939"} err="failed to get container status \"88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939\": rpc error: code = NotFound desc = could not find container \"88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939\": container with ID starting with 88cd8937fef47c9b7bebcb59d678367bec22995fae40fdc6fee011b6bdd45939 not found: ID does not exist" Oct 30 04:17:42.147912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.147747 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-socket\") pod \"ba448995-8974-4206-88a6-2c4f0259759f\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " Oct 30 04:17:42.147912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.147795 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-envoy\") pod \"ba448995-8974-4206-88a6-2c4f0259759f\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " Oct 30 04:17:42.147912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.147833 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-certs\") pod \"ba448995-8974-4206-88a6-2c4f0259759f\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " Oct 30 04:17:42.147912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.147894 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-istio-token\" (UniqueName: \"kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-istio-token\") pod \"ba448995-8974-4206-88a6-2c4f0259759f\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " Oct 30 04:17:42.148167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.147939 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba448995-8974-4206-88a6-2c4f0259759f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba448995-8974-4206-88a6-2c4f0259759f-istiod-ca-cert\") pod \"ba448995-8974-4206-88a6-2c4f0259759f\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " Oct 30 04:17:42.148167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.147966 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba448995-8974-4206-88a6-2c4f0259759f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba448995-8974-4206-88a6-2c4f0259759f-istio-podinfo\") pod \"ba448995-8974-4206-88a6-2c4f0259759f\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " Oct 30 04:17:42.148167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.147987 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-data\") pod \"ba448995-8974-4206-88a6-2c4f0259759f\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " Oct 30 04:17:42.148167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.148019 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-kube-api-access-8dwk2\" (UniqueName: \"kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-kube-api-access-8dwk2\") pod \"ba448995-8974-4206-88a6-2c4f0259759f\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " Oct 30 04:17:42.148167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.148051 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-credential-socket\") pod \"ba448995-8974-4206-88a6-2c4f0259759f\" (UID: \"ba448995-8974-4206-88a6-2c4f0259759f\") " Oct 30 04:17:42.148167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.148092 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-socket" pod "ba448995-8974-4206-88a6-2c4f0259759f" (UID: "ba448995-8974-4206-88a6-2c4f0259759f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:42.148512 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.148449 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:42.148951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.148910 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba448995-8974-4206-88a6-2c4f0259759f-istiod-ca-cert" pod "ba448995-8974-4206-88a6-2c4f0259759f" (UID: "ba448995-8974-4206-88a6-2c4f0259759f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:17:42.149622 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.149594 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-certs" pod "ba448995-8974-4206-88a6-2c4f0259759f" (UID: "ba448995-8974-4206-88a6-2c4f0259759f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:42.149787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.149764 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-data" pod "ba448995-8974-4206-88a6-2c4f0259759f" (UID: "ba448995-8974-4206-88a6-2c4f0259759f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:42.150014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.149991 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-credential-socket" pod "ba448995-8974-4206-88a6-2c4f0259759f" (UID: "ba448995-8974-4206-88a6-2c4f0259759f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:42.151971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.151944 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba448995-8974-4206-88a6-2c4f0259759f-istio-podinfo" pod "ba448995-8974-4206-88a6-2c4f0259759f" (UID: "ba448995-8974-4206-88a6-2c4f0259759f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:17:42.152071 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.152007 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-envoy" pod "ba448995-8974-4206-88a6-2c4f0259759f" (UID: "ba448995-8974-4206-88a6-2c4f0259759f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:17:42.152255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.152237 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-istio-token" pod "ba448995-8974-4206-88a6-2c4f0259759f" (UID: "ba448995-8974-4206-88a6-2c4f0259759f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:42.174440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.172560 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-kube-api-access-8dwk2" pod "ba448995-8974-4206-88a6-2c4f0259759f" (UID: "ba448995-8974-4206-88a6-2c4f0259759f"). InnerVolumeSpecName "kube-api-access-8dwk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:17:42.186437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.185691 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" podStartSLOduration=2.185678511 podStartE2EDuration="2.185678511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:42.184779428 +0000 UTC m=+1185.470654230" watchObservedRunningTime="2026-10-30 04:17:42.185678511 +0000 UTC m=+1185.471553286" Oct 30 04:17:42.209832 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.209771 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:42.249242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.249221 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:42.249242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.249243 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:42.249363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.249255 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:42.249363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.249263 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba448995-8974-4206-88a6-2c4f0259759f-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:42.249363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.249271 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba448995-8974-4206-88a6-2c4f0259759f-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:42.249363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.249278 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:42.249363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.249289 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dwk2\" (UniqueName: \"kubernetes.io/projected/ba448995-8974-4206-88a6-2c4f0259759f-kube-api-access-8dwk2\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:42.249363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.249298 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba448995-8974-4206-88a6-2c4f0259759f-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:17:42.285671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.285638 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:42.483521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.471270 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h"] Oct 30 04:17:42.485193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.485163 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-cb1a18be-openshift-default-5f858dbfb5-89s9h"] Oct 30 04:17:42.846567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.846483 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba448995-8974-4206-88a6-2c4f0259759f" path="/var/lib/kubelet/pods/ba448995-8974-4206-88a6-2c4f0259759f/volumes" Oct 30 04:17:42.999215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.999186 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" Oct 30 04:17:42.999456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:42.999411 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" Oct 30 04:17:43.894305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:43.894274 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:43.993676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:43.993648 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778"] Oct 30 04:17:43.994344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:43.994327 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" Oct 30 04:17:43.994407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:43.994348 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" Oct 30 04:17:43.994407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:43.994367 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" Oct 30 04:17:43.994407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:43.994376 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" Oct 30 04:17:43.994552 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:43.994503 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" Oct 30 04:17:43.998247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:43.998226 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" Oct 30 04:17:44.016854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.016827 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778"] Oct 30 04:17:44.025713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.025685 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-6796b1-dockercfg-md6bx\"" Oct 30 04:17:44.025867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.025799 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-6796b1\"" Oct 30 04:17:44.111445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.108932 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8"] Oct 30 04:17:44.111445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.109788 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba448995-8974-4206-88a6-2c4f0259759f" containerName="istio-proxy" Oct 30 04:17:44.111445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.110162 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvxq\" (UniqueName: \"kubernetes.io/projected/3698383d-bb0c-41c5-84d2-1b5aa813151e-kube-api-access-psvxq\") pod \"payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778\" (UID: \"3698383d-bb0c-41c5-84d2-1b5aa813151e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" Oct 30 04:17:44.111445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.110274 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3698383d-bb0c-41c5-84d2-1b5aa813151e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778\" (UID: \"3698383d-bb0c-41c5-84d2-1b5aa813151e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" Oct 30 04:17:44.114480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.113829 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:44.114862 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.114835 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8"] Oct 30 04:17:44.114980 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.114949 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" Oct 30 04:17:44.211623 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.211597 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3698383d-bb0c-41c5-84d2-1b5aa813151e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778\" (UID: \"3698383d-bb0c-41c5-84d2-1b5aa813151e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" Oct 30 04:17:44.211810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.211729 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b7ca3f-50a5-4aa2-9b53-241419d21cda-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8\" (UID: \"11b7ca3f-50a5-4aa2-9b53-241419d21cda\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" Oct 30 04:17:44.211810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.211763 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvxq\" (UniqueName: \"kubernetes.io/projected/3698383d-bb0c-41c5-84d2-1b5aa813151e-kube-api-access-psvxq\") pod \"payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778\" (UID: \"3698383d-bb0c-41c5-84d2-1b5aa813151e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" Oct 30 04:17:44.211933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.211889 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx7t4\" (UniqueName: \"kubernetes.io/projected/11b7ca3f-50a5-4aa2-9b53-241419d21cda-kube-api-access-zx7t4\") pod \"payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8\" (UID: \"11b7ca3f-50a5-4aa2-9b53-241419d21cda\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" Oct 30 04:17:44.212316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.212297 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3698383d-bb0c-41c5-84d2-1b5aa813151e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778\" (UID: \"3698383d-bb0c-41c5-84d2-1b5aa813151e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" Oct 30 04:17:44.236063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.236039 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvxq\" (UniqueName: \"kubernetes.io/projected/3698383d-bb0c-41c5-84d2-1b5aa813151e-kube-api-access-psvxq\") pod \"payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778\" (UID: \"3698383d-bb0c-41c5-84d2-1b5aa813151e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" Oct 30 04:17:44.311265 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.311225 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:44.312317 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.312099 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" Oct 30 04:17:44.312696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.312658 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b7ca3f-50a5-4aa2-9b53-241419d21cda-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8\" (UID: \"11b7ca3f-50a5-4aa2-9b53-241419d21cda\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" Oct 30 04:17:44.312838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.312722 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx7t4\" (UniqueName: \"kubernetes.io/projected/11b7ca3f-50a5-4aa2-9b53-241419d21cda-kube-api-access-zx7t4\") pod \"payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8\" (UID: \"11b7ca3f-50a5-4aa2-9b53-241419d21cda\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" Oct 30 04:17:44.313469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.313446 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b7ca3f-50a5-4aa2-9b53-241419d21cda-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8\" (UID: \"11b7ca3f-50a5-4aa2-9b53-241419d21cda\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" Oct 30 04:17:44.336363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.336315 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx7t4\" (UniqueName: \"kubernetes.io/projected/11b7ca3f-50a5-4aa2-9b53-241419d21cda-kube-api-access-zx7t4\") pod \"payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8\" (UID: \"11b7ca3f-50a5-4aa2-9b53-241419d21cda\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" Oct 30 04:17:44.434582 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.434548 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" Oct 30 04:17:44.616320 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.616294 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778"] Oct 30 04:17:44.618412 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:44.618384 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3698383d_bb0c_41c5_84d2_1b5aa813151e.slice/crio-4eda5c96f34b8f74e1edc250321b56acae026f049781fc65710342c26c90a037 WatchSource:0}: Error finding container 4eda5c96f34b8f74e1edc250321b56acae026f049781fc65710342c26c90a037: Status 404 returned error can't find the container with id 4eda5c96f34b8f74e1edc250321b56acae026f049781fc65710342c26c90a037 Oct 30 04:17:44.749729 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:44.749700 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8"] Oct 30 04:17:44.750497 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:44.750400 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b7ca3f_50a5_4aa2_9b53_241419d21cda.slice/crio-c89d1aef297b2d473ea47fffd341e7de1479b607decf6ecf665349a214d9a193 WatchSource:0}: Error finding container c89d1aef297b2d473ea47fffd341e7de1479b607decf6ecf665349a214d9a193: Status 404 returned error can't find the container with id c89d1aef297b2d473ea47fffd341e7de1479b607decf6ecf665349a214d9a193 Oct 30 04:17:45.143762 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:45.143677 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:45.148707 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:45.148678 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" event={"ID":"3698383d-bb0c-41c5-84d2-1b5aa813151e","Type":"ContainerStarted","Data":"2e37f499449e1bbf4361819b8f340524eff1ae9d7350b9cc8a7a3820fa838f85"} Oct 30 04:17:45.148871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:45.148741 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" Oct 30 04:17:45.148871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:45.148766 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" event={"ID":"3698383d-bb0c-41c5-84d2-1b5aa813151e","Type":"ContainerStarted","Data":"4eda5c96f34b8f74e1edc250321b56acae026f049781fc65710342c26c90a037"} Oct 30 04:17:45.150470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:45.150440 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" event={"ID":"11b7ca3f-50a5-4aa2-9b53-241419d21cda","Type":"ContainerStarted","Data":"527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c"} Oct 30 04:17:45.150609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:45.150488 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" event={"ID":"11b7ca3f-50a5-4aa2-9b53-241419d21cda","Type":"ContainerStarted","Data":"c89d1aef297b2d473ea47fffd341e7de1479b607decf6ecf665349a214d9a193"} Oct 30 04:17:45.150609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:45.150524 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" Oct 30 04:17:45.194020 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:45.193956 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" podStartSLOduration=2.193939701 podStartE2EDuration="2.193939701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:45.19389054 +0000 UTC m=+1188.479765316" watchObservedRunningTime="2026-10-30 04:17:45.193939701 +0000 UTC m=+1188.479814477" Oct 30 04:17:45.238176 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:45.238103 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" podStartSLOduration=1.238087242 podStartE2EDuration="1.238087242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:45.234691866 +0000 UTC m=+1188.520566641" watchObservedRunningTime="2026-10-30 04:17:45.238087242 +0000 UTC m=+1188.523962017" Oct 30 04:17:45.786508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:45.786465 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:46.312646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:46.312601 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:46.544395 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:46.544334 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:46.845762 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:46.845717 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:47.030808 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:47.030779 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" Oct 30 04:17:47.030971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:47.030913 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" Oct 30 04:17:47.399620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:47.399579 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:47.622673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:47.622587 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:48.047302 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:48.047268 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:48.173870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:48.173838 4793 generic.go:356] "Generic (PLEG): container finished" podID="2da898aa-742b-41a7-9c66-ab97460daf99" containerID="60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9" exitCode=0 Oct 30 04:17:48.174036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:48.173917 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" event={"ID":"2da898aa-742b-41a7-9c66-ab97460daf99","Type":"ContainerDied","Data":"60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9"} Oct 30 04:17:48.174363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:48.174347 4793 scope.go:122] "RemoveContainer" containerID="60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9" Oct 30 04:17:48.175804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:48.175778 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:48.175882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:48.175845 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:48.175882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:48.175877 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:17:49.201470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:49.201433 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" event={"ID":"2da898aa-742b-41a7-9c66-ab97460daf99","Type":"ContainerStarted","Data":"eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b"} Oct 30 04:17:49.825254 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:49.825203 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:50.135037 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.134949 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:50.135212 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.135155 4793 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 04:17:50.135275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.135208 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 04:17:50.504466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.504432 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz"] Oct 30 04:17:50.512860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.512835 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.528115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.528090 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz"] Oct 30 04:17:50.536943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.536919 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b4708a\"/\"openshift-service-ca.crt\"" Oct 30 04:17:50.536943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.536932 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b4708a\"/\"kube-root-ca.crt\"" Oct 30 04:17:50.540511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.540485 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b4708a\"/\"simulator-w3-b4708a-kserve-self-signed-certs\"" Oct 30 04:17:50.540609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.540552 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b4708a\"/\"default-dockercfg-6p4tv\"" Oct 30 04:17:50.610988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.610957 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbvwq\" (UniqueName: \"kubernetes.io/projected/c9e51d14-8150-4f8c-9172-845b3b99eca9-kube-api-access-kbvwq\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.611131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.610999 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-tmp-dir\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.611131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.611024 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-dshm\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.611205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.611128 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-model-cache\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.611205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.611167 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-home\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.611286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.611267 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e51d14-8150-4f8c-9172-845b3b99eca9-tls-certs\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.652398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.652372 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:50.712607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.712578 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-tmp-dir\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.712756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.712618 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-dshm\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.712756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.712665 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-model-cache\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.712756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.712700 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-home\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.712756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.712754 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e51d14-8150-4f8c-9172-845b3b99eca9-tls-certs\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.712978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.712846 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbvwq\" (UniqueName: \"kubernetes.io/projected/c9e51d14-8150-4f8c-9172-845b3b99eca9-kube-api-access-kbvwq\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.712978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.712952 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-tmp-dir\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.713092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.713050 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-model-cache\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.713165 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.713144 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-home\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.714742 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.714715 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-dshm\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.715071 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.715051 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e51d14-8150-4f8c-9172-845b3b99eca9-tls-certs\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.744017 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.743952 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbvwq\" (UniqueName: \"kubernetes.io/projected/c9e51d14-8150-4f8c-9172-845b3b99eca9-kube-api-access-kbvwq\") pod \"simulator-w3-b4708a-kserve-57bf46994d-28mgz\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:50.826067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:50.825989 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:51.076547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:51.076390 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:51.135128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:51.135096 4793 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 04:17:51.135290 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:51.135160 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 04:17:51.180310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:51.180236 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz"] Oct 30 04:17:51.186303 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:51.186272 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9e51d14_8150_4f8c_9172_845b3b99eca9.slice/crio-1f682632df335aa9cbe0e6e3337f8a3e3ec4c09555e67cdfe7d1c436bc0b747a WatchSource:0}: Error finding container 1f682632df335aa9cbe0e6e3337f8a3e3ec4c09555e67cdfe7d1c436bc0b747a: Status 404 returned error can't find the container with id 1f682632df335aa9cbe0e6e3337f8a3e3ec4c09555e67cdfe7d1c436bc0b747a Oct 30 04:17:51.217222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:51.217189 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" event={"ID":"c9e51d14-8150-4f8c-9172-845b3b99eca9","Type":"ContainerStarted","Data":"1f682632df335aa9cbe0e6e3337f8a3e3ec4c09555e67cdfe7d1c436bc0b747a"} Oct 30 04:17:51.287088 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:51.287042 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:51.300674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:51.300259 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:51.848969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:51.848928 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:52.084996 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:52.084959 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:52.135394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:52.135322 4793 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 04:17:52.135394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:52.135371 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 04:17:52.224732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:52.224695 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" event={"ID":"c9e51d14-8150-4f8c-9172-845b3b99eca9","Type":"ContainerStarted","Data":"51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f"} Oct 30 04:17:52.236335 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:52.236309 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 04:17:52.282795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:52.282715 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" podStartSLOduration=2.282695791 podStartE2EDuration="2.282695791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:52.273681185 +0000 UTC m=+1195.559555960" watchObservedRunningTime="2026-10-30 04:17:52.282695791 +0000 UTC m=+1195.568570569" Oct 30 04:17:52.306318 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:52.306287 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:52.826323 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:52.826291 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:17:53.134834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:53.134748 4793 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 04:17:53.134834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:53.134801 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 04:17:53.892485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:53.892453 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:54.135310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:54.135269 4793 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Oct 30 04:17:54.135759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:54.135347 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Oct 30 04:17:54.917885 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:54.917842 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:55.138687 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.138657 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:55.248480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.248447 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 04:17:55.619340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.619252 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:55.795638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.795603 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc"] Oct 30 04:17:55.803756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.803729 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.818602 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.818562 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc"] Oct 30 04:17:55.819387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.819263 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a2f984\"/\"default-dockercfg-sqvb6\"" Oct 30 04:17:55.819387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.819266 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a2f984\"/\"kube-root-ca.crt\"" Oct 30 04:17:55.819387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.819301 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-a2f984\"/\"openshift-service-ca.crt\"" Oct 30 04:17:55.828461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.828442 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-a2f984\"/\"simulator-w4-a2f984-kserve-self-signed-certs\"" Oct 30 04:17:55.880708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.880637 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7wfb\" (UniqueName: \"kubernetes.io/projected/d1176b48-69f0-47e0-9de0-da5d70421762-kube-api-access-n7wfb\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.880708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.880682 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-dshm\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.880993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.880972 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1176b48-69f0-47e0-9de0-da5d70421762-tls-certs\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.881103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.881008 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-home\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.881103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.881089 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-tmp-dir\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.881228 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.881190 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-model-cache\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.952140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.952101 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:55.982658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.982634 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1176b48-69f0-47e0-9de0-da5d70421762-tls-certs\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.982765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.982666 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-home\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.982765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.982689 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-tmp-dir\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.982765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.982722 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-model-cache\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.982880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.982858 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7wfb\" (UniqueName: \"kubernetes.io/projected/d1176b48-69f0-47e0-9de0-da5d70421762-kube-api-access-n7wfb\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.982917 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.982892 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-dshm\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.983088 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.983063 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-home\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.983203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.983142 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-tmp-dir\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.983254 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.983206 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-model-cache\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.984994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.984972 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-dshm\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:55.985249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:55.985232 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1176b48-69f0-47e0-9de0-da5d70421762-tls-certs\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:56.006871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:56.006844 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7wfb\" (UniqueName: \"kubernetes.io/projected/d1176b48-69f0-47e0-9de0-da5d70421762-kube-api-access-n7wfb\") pod \"simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:56.116933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:56.116900 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:56.162388 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:56.161402 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" Oct 30 04:17:56.163465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:56.163247 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" Oct 30 04:17:56.434299 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:56.434274 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc"] Oct 30 04:17:56.436490 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:17:56.436460 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1176b48_69f0_47e0_9de0_da5d70421762.slice/crio-d9ff136bb07fd202bd407349fb777634e47834ded314cda61792c59f2beaf5cd WatchSource:0}: Error finding container d9ff136bb07fd202bd407349fb777634e47834ded314cda61792c59f2beaf5cd: Status 404 returned error can't find the container with id d9ff136bb07fd202bd407349fb777634e47834ded314cda61792c59f2beaf5cd Oct 30 04:17:57.285966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:57.285865 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" event={"ID":"d1176b48-69f0-47e0-9de0-da5d70421762","Type":"ContainerStarted","Data":"de3b9a159091f3e54ec6f2adc6f53d493e5e80f28420236644ff4bb04a8d5f5a"} Oct 30 04:17:57.285966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:57.285920 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" event={"ID":"d1176b48-69f0-47e0-9de0-da5d70421762","Type":"ContainerStarted","Data":"d9ff136bb07fd202bd407349fb777634e47834ded314cda61792c59f2beaf5cd"} Oct 30 04:17:57.386458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:57.386390 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" podStartSLOduration=2.386372574 podStartE2EDuration="2.386372574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:17:57.38393175 +0000 UTC m=+1200.669806555" watchObservedRunningTime="2026-10-30 04:17:57.386372574 +0000 UTC m=+1200.672247348" Oct 30 04:17:57.491749 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:57.491704 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:17:58.118004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:58.117971 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:17:59.236255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:17:59.236211 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:00.826653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:00.826621 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:18:00.846064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:00.846034 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:18:00.900491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:00.900411 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:01.328518 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:01.328489 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 04:18:01.939070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:01.939034 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:02.631458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:02.630560 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:04.428938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:04.428898 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:05.351426 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:05.351385 4793 generic.go:356] "Generic (PLEG): container finished" podID="0886d884-9043-4114-ba66-829bf6f319b7" containerID="e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a" exitCode=0 Oct 30 04:18:05.351667 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:05.351463 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" event={"ID":"0886d884-9043-4114-ba66-829bf6f319b7","Type":"ContainerDied","Data":"e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a"} Oct 30 04:18:05.351933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:05.351915 4793 scope.go:122] "RemoveContainer" containerID="e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a" Oct 30 04:18:05.353230 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:05.353201 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:18:05.353283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:05.353270 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:18:05.353318 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:05.353301 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:18:06.117169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:06.117123 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:18:06.134780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:06.134751 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:18:06.384549 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:06.382184 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" event={"ID":"0886d884-9043-4114-ba66-829bf6f319b7","Type":"ContainerStarted","Data":"78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12"} Oct 30 04:18:06.384549 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:06.384230 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:06.396194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:06.396167 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 04:18:06.891092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:06.890924 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:07.053086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:07.053036 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:18:07.064439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:07.063542 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 04:18:07.064439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:07.063616 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 04:18:07.582442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:07.581552 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:07.801856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:07.801811 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:08.053067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:08.053032 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 04:18:08.053246 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:08.053097 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 04:18:08.147040 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:08.147004 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:09.068443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:09.066267 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 04:18:09.068443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:09.066337 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 04:18:09.808167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:09.807964 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:10.027610 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:10.027570 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:10.053751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:10.053719 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 04:18:10.054004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:10.053978 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 04:18:10.106568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:10.106494 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:18:11.054552 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:11.054515 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 04:18:11.054823 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:11.054795 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 04:18:11.220531 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:18:11.220492 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:18:11.702955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:11.702916 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:12.057432 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:12.057345 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:18:12.058456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:12.058409 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:18:13.421160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:13.421123 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:15.118969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:15.118928 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:16.836218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:16.836172 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:17.272328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.272292 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f"] Oct 30 04:18:17.277031 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.277005 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.304912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.304884 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b4708a\"/\"premium-w3-b4708a-kserve-self-signed-certs\"" Oct 30 04:18:17.309304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.309273 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f"] Oct 30 04:18:17.414887 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.414836 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-dshm\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.414887 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.414890 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-home\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.415137 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.414980 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53f3359-41ba-40d7-9820-c2409a0a6753-tls-certs\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.415137 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.415013 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-model-cache\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.415253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.415193 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-tmp-dir\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.415308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.415253 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn22v\" (UniqueName: \"kubernetes.io/projected/f53f3359-41ba-40d7-9820-c2409a0a6753-kube-api-access-dn22v\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.436124 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.436087 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:17.516380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.516347 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-tmp-dir\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.516569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.516394 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn22v\" (UniqueName: \"kubernetes.io/projected/f53f3359-41ba-40d7-9820-c2409a0a6753-kube-api-access-dn22v\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.516569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.516538 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-dshm\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.516658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.516582 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-home\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.516658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.516625 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53f3359-41ba-40d7-9820-c2409a0a6753-tls-certs\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.516752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.516661 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-model-cache\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.516838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.516810 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-tmp-dir\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.517013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.516988 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-home\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.517092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.517025 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-model-cache\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.518760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.518731 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-dshm\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.519016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.518998 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53f3359-41ba-40d7-9820-c2409a0a6753-tls-certs\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.539728 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.539670 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn22v\" (UniqueName: \"kubernetes.io/projected/f53f3359-41ba-40d7-9820-c2409a0a6753-kube-api-access-dn22v\") pod \"premium-w3-b4708a-kserve-5c58db468f-pg86f\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.593372 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.593340 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:17.893840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:17.893804 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f"] Oct 30 04:18:17.894197 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:18:17.893916 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf53f3359_41ba_40d7_9820_c2409a0a6753.slice/crio-42ce81d2531826c403cb622d930aad8d352e5af9bd167c53e6d5609afa05de4b WatchSource:0}: Error finding container 42ce81d2531826c403cb622d930aad8d352e5af9bd167c53e6d5609afa05de4b: Status 404 returned error can't find the container with id 42ce81d2531826c403cb622d930aad8d352e5af9bd167c53e6d5609afa05de4b Oct 30 04:18:18.464721 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:18.464681 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:18.479065 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:18.479032 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" event={"ID":"f53f3359-41ba-40d7-9820-c2409a0a6753","Type":"ContainerStarted","Data":"86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228"} Oct 30 04:18:18.479238 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:18.479071 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" event={"ID":"f53f3359-41ba-40d7-9820-c2409a0a6753","Type":"ContainerStarted","Data":"42ce81d2531826c403cb622d930aad8d352e5af9bd167c53e6d5609afa05de4b"} Oct 30 04:18:18.520472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:18.520390 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" podStartSLOduration=1.520372289 podStartE2EDuration="1.520372289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:18.520222237 +0000 UTC m=+1221.806097013" watchObservedRunningTime="2026-10-30 04:18:18.520372289 +0000 UTC m=+1221.806247065" Oct 30 04:18:18.593462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:18.593436 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:20.207694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:20.207644 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:21.656014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:21.655971 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:22.021829 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:22.021798 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:23.712250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:23.712217 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:25.325809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:25.325772 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:27.104957 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:27.104925 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:27.611674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:27.609780 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:27.632135 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:27.632103 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:28.555147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:28.555117 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 04:18:28.900451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:28.900324 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:30.174299 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:30.174254 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:30.719286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:30.719258 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:32.338004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:32.337970 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:34.049244 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:34.049204 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:35.506119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.506082 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz"] Oct 30 04:18:35.511156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.511134 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.526480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.526457 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5d74b9-openshift-default-dockercfg-d8777\"" Oct 30 04:18:35.543525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.543500 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz"] Oct 30 04:18:35.611701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.611673 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.611852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.611719 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.611852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.611831 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.611978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.611869 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.612033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.612011 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.612093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.612048 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.612192 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.612166 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgfdn\" (UniqueName: \"kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-kube-api-access-wgfdn\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.612246 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.612229 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.612296 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.612281 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.713861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.713830 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgfdn\" (UniqueName: \"kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-kube-api-access-wgfdn\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.713874 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.713915 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.713958 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.714074 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.714136 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.714176 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.714284 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.714321 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714507 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.714488 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.714528 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.714580 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714720 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.714699 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.714797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.714774 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.716502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.716473 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.716598 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.716564 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.737564 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.737543 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgfdn\" (UniqueName: \"kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-kube-api-access-wgfdn\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.737680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.737620 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.825056 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.824968 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:35.836500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:35.836389 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:36.147306 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:36.147282 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz"] Oct 30 04:18:36.148526 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:18:36.148496 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b29e1f_12ff_4b1a_8fbc_52e926ce5ba4.slice/crio-458b38f38e4742adf767a1b8f0df0912e6f13276ee05b3591a96bf1c070b2ee8 WatchSource:0}: Error finding container 458b38f38e4742adf767a1b8f0df0912e6f13276ee05b3591a96bf1c070b2ee8: Status 404 returned error can't find the container with id 458b38f38e4742adf767a1b8f0df0912e6f13276ee05b3591a96bf1c070b2ee8 Oct 30 04:18:36.150898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:36.150863 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:18:36.150971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:36.150931 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:18:36.150971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:36.150962 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:18:36.598722 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:36.598688 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" event={"ID":"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4","Type":"ContainerStarted","Data":"1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc"} Oct 30 04:18:36.599131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:36.598743 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" event={"ID":"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4","Type":"ContainerStarted","Data":"458b38f38e4742adf767a1b8f0df0912e6f13276ee05b3591a96bf1c070b2ee8"} Oct 30 04:18:36.657686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:36.657633 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" podStartSLOduration=1.6576190020000001 podStartE2EDuration="1.657619002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:36.656321187 +0000 UTC m=+1239.942195963" watchObservedRunningTime="2026-10-30 04:18:36.657619002 +0000 UTC m=+1239.943493777" Oct 30 04:18:36.826063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:36.826033 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:37.379440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:37.376948 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:37.589287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:37.589253 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:37.827997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:37.827962 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 04:18:37.828380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:37.828015 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" podUID="e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 04:18:38.826129 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:38.826094 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 04:18:38.826307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:38.826147 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" podUID="e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 04:18:39.217390 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:39.217357 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:39.826074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:39.826039 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 04:18:39.826256 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:39.826095 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" podUID="e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 04:18:40.825671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:40.825641 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 04:18:40.826100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:40.825695 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" podUID="e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 04:18:40.826649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:40.826616 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:41.219700 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:18:41.219660 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:18:41.826490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:41.826455 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 04:18:41.826922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:41.826507 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" podUID="e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 04:18:42.598306 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:42.598268 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:43.053921 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:43.053883 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:43.055710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:43.055678 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:43.056741 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:43.056707 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 04:18:44.346165 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:44.346118 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:46.318774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:46.318746 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:48.030770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:48.030724 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:49.624660 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:49.624624 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:49.858705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:49.858663 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:51.727774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:51.727737 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:52.341835 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:52.341796 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:53.597290 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.597247 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:53.793748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.793719 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx"] Oct 30 04:18:53.797849 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.797830 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:53.816372 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.816349 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx"] Oct 30 04:18:53.824711 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.824683 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:18:53.824805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.824709 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:18:53.824805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.824720 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:18:53.825752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.825732 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:18:53.837384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.837356 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef178ec8-2wjvx\" (UID: \"b0d68363-4dde-45c1-91e0-d9bde42a703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:53.837509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.837408 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef178ec8-2wjvx\" (UID: \"b0d68363-4dde-45c1-91e0-d9bde42a703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:53.837710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.837691 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qqkm\" (UniqueName: \"kubernetes.io/projected/b0d68363-4dde-45c1-91e0-d9bde42a703a-kube-api-access-9qqkm\") pod \"maas-api-revoke-sub-ef178ec8-2wjvx\" (UID: \"b0d68363-4dde-45c1-91e0-d9bde42a703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:53.938678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.938647 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qqkm\" (UniqueName: \"kubernetes.io/projected/b0d68363-4dde-45c1-91e0-d9bde42a703a-kube-api-access-9qqkm\") pod \"maas-api-revoke-sub-ef178ec8-2wjvx\" (UID: \"b0d68363-4dde-45c1-91e0-d9bde42a703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:53.938859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.938708 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef178ec8-2wjvx\" (UID: \"b0d68363-4dde-45c1-91e0-d9bde42a703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:53.938859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.938728 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef178ec8-2wjvx\" (UID: \"b0d68363-4dde-45c1-91e0-d9bde42a703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:53.939313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.939289 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef178ec8-2wjvx\" (UID: \"b0d68363-4dde-45c1-91e0-d9bde42a703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:53.941004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.940987 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef178ec8-2wjvx\" (UID: \"b0d68363-4dde-45c1-91e0-d9bde42a703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:53.962536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:53.962517 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qqkm\" (UniqueName: \"kubernetes.io/projected/b0d68363-4dde-45c1-91e0-d9bde42a703a-kube-api-access-9qqkm\") pod \"maas-api-revoke-sub-ef178ec8-2wjvx\" (UID: \"b0d68363-4dde-45c1-91e0-d9bde42a703a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:54.112095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:54.112064 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:54.417610 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:54.417586 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx"] Oct 30 04:18:54.418684 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:18:54.418649 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d68363_4dde_45c1_91e0_d9bde42a703a.slice/crio-17b0ac27387eb2bb970bbdead1cc38dd6810de1c60d5fb3c1350e49904b855ff WatchSource:0}: Error finding container 17b0ac27387eb2bb970bbdead1cc38dd6810de1c60d5fb3c1350e49904b855ff: Status 404 returned error can't find the container with id 17b0ac27387eb2bb970bbdead1cc38dd6810de1c60d5fb3c1350e49904b855ff Oct 30 04:18:54.719885 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:54.719850 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" event={"ID":"b0d68363-4dde-45c1-91e0-d9bde42a703a","Type":"ContainerStarted","Data":"17b0ac27387eb2bb970bbdead1cc38dd6810de1c60d5fb3c1350e49904b855ff"} Oct 30 04:18:55.223963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:55.223926 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:55.727005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:55.726972 4793 generic.go:356] "Generic (PLEG): container finished" podID="b0d68363-4dde-45c1-91e0-d9bde42a703a" containerID="fa4367f605055f431c6a1a79b0fc51225f04f4fb298bc528603529d2bc544356" exitCode=0 Oct 30 04:18:55.727355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:55.727055 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" event={"ID":"b0d68363-4dde-45c1-91e0-d9bde42a703a","Type":"ContainerDied","Data":"fa4367f605055f431c6a1a79b0fc51225f04f4fb298bc528603529d2bc544356"} Oct 30 04:18:56.875824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:56.875799 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:56.973625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:56.973595 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-xks-ca\") pod \"b0d68363-4dde-45c1-91e0-d9bde42a703a\" (UID: \"b0d68363-4dde-45c1-91e0-d9bde42a703a\") " Oct 30 04:18:56.973805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:56.973787 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d68363-4dde-45c1-91e0-d9bde42a703a-kube-api-access-9qqkm\" (UniqueName: \"kubernetes.io/projected/b0d68363-4dde-45c1-91e0-d9bde42a703a-kube-api-access-9qqkm\") pod \"b0d68363-4dde-45c1-91e0-d9bde42a703a\" (UID: \"b0d68363-4dde-45c1-91e0-d9bde42a703a\") " Oct 30 04:18:56.973899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:56.973883 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-openshift-ca\") pod \"b0d68363-4dde-45c1-91e0-d9bde42a703a\" (UID: \"b0d68363-4dde-45c1-91e0-d9bde42a703a\") " Oct 30 04:18:56.974227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:56.974192 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-openshift-ca" pod "b0d68363-4dde-45c1-91e0-d9bde42a703a" (UID: "b0d68363-4dde-45c1-91e0-d9bde42a703a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:18:56.975743 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:56.975712 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-xks-ca" pod "b0d68363-4dde-45c1-91e0-d9bde42a703a" (UID: "b0d68363-4dde-45c1-91e0-d9bde42a703a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:18:56.975896 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:56.975870 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d68363-4dde-45c1-91e0-d9bde42a703a-kube-api-access-9qqkm" pod "b0d68363-4dde-45c1-91e0-d9bde42a703a" (UID: "b0d68363-4dde-45c1-91e0-d9bde42a703a"). InnerVolumeSpecName "kube-api-access-9qqkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:18:57.001911 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.001838 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:57.075789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.075756 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qqkm\" (UniqueName: \"kubernetes.io/projected/b0d68363-4dde-45c1-91e0-d9bde42a703a-kube-api-access-9qqkm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:18:57.075789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.075787 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:18:57.075789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.075799 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0d68363-4dde-45c1-91e0-d9bde42a703a-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:18:57.507360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.507329 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s"] Oct 30 04:18:57.507877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.507861 4793 cpu_manager.go:414] "containerMap: removing container" podUID="b0d68363-4dde-45c1-91e0-d9bde42a703a" containerName="revoke-keys" Oct 30 04:18:57.507941 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.507879 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d68363-4dde-45c1-91e0-d9bde42a703a" containerName="revoke-keys" Oct 30 04:18:57.507986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.507963 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d68363-4dde-45c1-91e0-d9bde42a703a" containerName="revoke-keys" Oct 30 04:18:57.511324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.511298 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.526457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.526437 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-36bbac-openshift-default-dockercfg-n6t7k\"" Oct 30 04:18:57.543972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.543942 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s"] Oct 30 04:18:57.581244 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.581207 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.581366 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.581267 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.581366 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.581341 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66jr8\" (UniqueName: \"kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-kube-api-access-66jr8\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.581516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.581394 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.581516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.581425 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.581516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.581460 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.581666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.581639 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.581712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.581684 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.581756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.581711 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.682303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.682276 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.682484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.682317 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66jr8\" (UniqueName: \"kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-kube-api-access-66jr8\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.682484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.682359 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.682484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.682377 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.682484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.682429 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.682701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.682505 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.682701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.682534 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.682701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.682558 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.682701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.682629 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.682890 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.682742 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.683100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.683041 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.683212 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.683099 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.683294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.683203 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.683466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.683403 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.685087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.685065 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.685340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.685322 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.705924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.705903 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66jr8\" (UniqueName: \"kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-kube-api-access-66jr8\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.706086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.706065 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:57.740608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.740582 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" Oct 30 04:18:57.740735 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.740608 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx" event={"ID":"b0d68363-4dde-45c1-91e0-d9bde42a703a","Type":"ContainerDied","Data":"17b0ac27387eb2bb970bbdead1cc38dd6810de1c60d5fb3c1350e49904b855ff"} Oct 30 04:18:57.740735 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.740652 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17b0ac27387eb2bb970bbdead1cc38dd6810de1c60d5fb3c1350e49904b855ff" Oct 30 04:18:57.825643 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:57.825584 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:58.149084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:58.149055 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s"] Oct 30 04:18:58.149538 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:18:58.149515 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41615d2b_3148_4a60_a6a0_4ad6191bbd4e.slice/crio-b2e2fed709f1b606937bbb6890a29cbe9bdbf28982b31db9a4cb959320a50f17 WatchSource:0}: Error finding container b2e2fed709f1b606937bbb6890a29cbe9bdbf28982b31db9a4cb959320a50f17: Status 404 returned error can't find the container with id b2e2fed709f1b606937bbb6890a29cbe9bdbf28982b31db9a4cb959320a50f17 Oct 30 04:18:58.152015 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:58.151982 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:18:58.152096 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:58.152045 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:18:58.152096 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:58.152072 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:18:58.710121 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:58.710078 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:18:58.749761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:58.749719 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" event={"ID":"41615d2b-3148-4a60-a6a0-4ad6191bbd4e","Type":"ContainerStarted","Data":"978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa"} Oct 30 04:18:58.749937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:58.749818 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" event={"ID":"41615d2b-3148-4a60-a6a0-4ad6191bbd4e","Type":"ContainerStarted","Data":"b2e2fed709f1b606937bbb6890a29cbe9bdbf28982b31db9a4cb959320a50f17"} Oct 30 04:18:58.804113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:58.804053 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" podStartSLOduration=1.804035493 podStartE2EDuration="1.804035493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:18:58.79938794 +0000 UTC m=+1262.085262737" watchObservedRunningTime="2026-10-30 04:18:58.804035493 +0000 UTC m=+1262.089910269" Oct 30 04:18:58.826406 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:58.826382 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:18:59.828533 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:59.828497 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 04:18:59.828933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:18:59.828555 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" podUID="41615d2b-3148-4a60-a6a0-4ad6191bbd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 04:19:00.337183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:00.337145 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:00.826714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:00.826681 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 04:19:00.826899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:00.826740 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" podUID="41615d2b-3148-4a60-a6a0-4ad6191bbd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 04:19:01.826674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:01.826639 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 04:19:01.827083 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:01.826685 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" podUID="41615d2b-3148-4a60-a6a0-4ad6191bbd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 04:19:01.926474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:01.926437 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:02.826018 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:02.825984 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 04:19:02.826168 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:02.826039 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" podUID="41615d2b-3148-4a60-a6a0-4ad6191bbd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 04:19:03.630446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:03.630381 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:03.829813 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:03.829770 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 04:19:03.830105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:03.830074 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" podUID="41615d2b-3148-4a60-a6a0-4ad6191bbd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 04:19:04.897443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:04.897392 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:19:04.897936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:04.897913 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:19:04.898648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:04.898626 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:19:05.618993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:05.618962 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:06.044699 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.044661 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv"] Oct 30 04:19:06.049459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.049435 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.068017 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.067992 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b4708a-openshift-default-dockercfg-t5rn6\"" Oct 30 04:19:06.080959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.080931 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv"] Oct 30 04:19:06.089215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.089188 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.089332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.089220 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkc7n\" (UniqueName: \"kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-kube-api-access-lkc7n\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.089332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.089316 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.089516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.089351 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.089516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.089378 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.089516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.089498 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.089685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.089532 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.089743 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.089690 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.089743 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.089727 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191168 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191137 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191168 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191170 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191208 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191225 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkc7n\" (UniqueName: \"kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-kube-api-access-lkc7n\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191337 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191382 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191410 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191480 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191538 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191703 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191754 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191929 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191817 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.191929 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.191848 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.192167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.192144 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.193788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.193766 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.193948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.193925 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.214512 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.214488 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.214884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.214867 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkc7n\" (UniqueName: \"kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-kube-api-access-lkc7n\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.365299 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.365212 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:06.693017 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.692988 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv"] Oct 30 04:19:06.693163 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:19:06.693054 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1b3d8c_6a80_400c_aebe_bbab2430d494.slice/crio-32be328ab27b8e60ec735af36eeab7f4609f2ceb5e04207d6c8ce315bbaee3d1 WatchSource:0}: Error finding container 32be328ab27b8e60ec735af36eeab7f4609f2ceb5e04207d6c8ce315bbaee3d1: Status 404 returned error can't find the container with id 32be328ab27b8e60ec735af36eeab7f4609f2ceb5e04207d6c8ce315bbaee3d1 Oct 30 04:19:06.695407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.695375 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:19:06.695567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.695473 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:19:06.695567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.695514 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:19:06.848992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.848955 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" event={"ID":"bc1b3d8c-6a80-400c-aebe-bbab2430d494","Type":"ContainerStarted","Data":"9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5"} Oct 30 04:19:06.849173 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.849012 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" event={"ID":"bc1b3d8c-6a80-400c-aebe-bbab2430d494","Type":"ContainerStarted","Data":"32be328ab27b8e60ec735af36eeab7f4609f2ceb5e04207d6c8ce315bbaee3d1"} Oct 30 04:19:06.899751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:06.899686 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" podStartSLOduration=0.899667633 podStartE2EDuration="899.667633ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:06.895841422 +0000 UTC m=+1270.181716438" watchObservedRunningTime="2026-10-30 04:19:06.899667633 +0000 UTC m=+1270.185542409" Oct 30 04:19:07.335092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:07.335056 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:07.384692 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:07.384658 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:07.386645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:07.386615 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 04:19:07.386787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:07.386673 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" podUID="bc1b3d8c-6a80-400c-aebe-bbab2430d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 04:19:08.378568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:08.378523 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 04:19:08.379024 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:08.378588 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" podUID="bc1b3d8c-6a80-400c-aebe-bbab2430d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 04:19:09.082445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.078055 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:09.091348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.091317 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq"] Oct 30 04:19:09.097220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.097182 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:09.114676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.114650 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq"] Oct 30 04:19:09.122734 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.122712 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:19:09.122734 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.122712 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:19:09.122909 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.122745 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:19:09.126292 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.126265 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f09c077-cg9sq\" (UID: \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:09.126384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.126299 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmrb5\" (UniqueName: \"kubernetes.io/projected/c0fe9fdc-4b19-494e-8c29-455263ac88f7-kube-api-access-nmrb5\") pod \"maas-api-revoke-sub-6f09c077-cg9sq\" (UID: \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:09.126524 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.126500 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f09c077-cg9sq\" (UID: \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:09.127172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.127150 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:19:09.227429 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.227383 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f09c077-cg9sq\" (UID: \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:09.227584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.227505 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f09c077-cg9sq\" (UID: \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:09.227584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.227535 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmrb5\" (UniqueName: \"kubernetes.io/projected/c0fe9fdc-4b19-494e-8c29-455263ac88f7-kube-api-access-nmrb5\") pod \"maas-api-revoke-sub-6f09c077-cg9sq\" (UID: \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:09.228240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.228207 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f09c077-cg9sq\" (UID: \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:09.229784 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.229759 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f09c077-cg9sq\" (UID: \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:09.251350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.251324 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmrb5\" (UniqueName: \"kubernetes.io/projected/c0fe9fdc-4b19-494e-8c29-455263ac88f7-kube-api-access-nmrb5\") pod \"maas-api-revoke-sub-6f09c077-cg9sq\" (UID: \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:09.366498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.365760 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 04:19:09.366498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.365834 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" podUID="bc1b3d8c-6a80-400c-aebe-bbab2430d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 04:19:09.416822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.416790 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:09.745981 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:19:09.745951 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fe9fdc_4b19_494e_8c29_455263ac88f7.slice/crio-53bcc8eef2f08deed3aaaf0662a755b7fa76fdc73fdf781ad9d2ba47ec1fef24 WatchSource:0}: Error finding container 53bcc8eef2f08deed3aaaf0662a755b7fa76fdc73fdf781ad9d2ba47ec1fef24: Status 404 returned error can't find the container with id 53bcc8eef2f08deed3aaaf0662a755b7fa76fdc73fdf781ad9d2ba47ec1fef24 Oct 30 04:19:09.747043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.747015 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq"] Oct 30 04:19:09.891361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:09.891324 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" event={"ID":"c0fe9fdc-4b19-494e-8c29-455263ac88f7","Type":"ContainerStarted","Data":"53bcc8eef2f08deed3aaaf0662a755b7fa76fdc73fdf781ad9d2ba47ec1fef24"} Oct 30 04:19:10.372502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:10.367777 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 04:19:10.372502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:10.367836 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" podUID="bc1b3d8c-6a80-400c-aebe-bbab2430d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 04:19:10.711455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:10.711411 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:10.899454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:10.899394 4793 generic.go:356] "Generic (PLEG): container finished" podID="c0fe9fdc-4b19-494e-8c29-455263ac88f7" containerID="581f480ed311b6188ad034ac7e793a68a79a9778fb68e6380a8314b46ddf3311" exitCode=0 Oct 30 04:19:10.899655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:10.899616 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" event={"ID":"c0fe9fdc-4b19-494e-8c29-455263ac88f7","Type":"ContainerDied","Data":"581f480ed311b6188ad034ac7e793a68a79a9778fb68e6380a8314b46ddf3311"} Oct 30 04:19:11.220100 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:19:11.220068 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:19:11.373442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:11.367708 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 04:19:11.373442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:11.367766 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" podUID="bc1b3d8c-6a80-400c-aebe-bbab2430d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 04:19:12.122089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.122064 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:12.302440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.293876 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-openshift-ca\") pod \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\" (UID: \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\") " Oct 30 04:19:12.302440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.293987 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fe9fdc-4b19-494e-8c29-455263ac88f7-kube-api-access-nmrb5\" (UniqueName: \"kubernetes.io/projected/c0fe9fdc-4b19-494e-8c29-455263ac88f7-kube-api-access-nmrb5\") pod \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\" (UID: \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\") " Oct 30 04:19:12.302440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.294087 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-xks-ca\") pod \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\" (UID: \"c0fe9fdc-4b19-494e-8c29-455263ac88f7\") " Oct 30 04:19:12.302440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.295156 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-openshift-ca" pod "c0fe9fdc-4b19-494e-8c29-455263ac88f7" (UID: "c0fe9fdc-4b19-494e-8c29-455263ac88f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:12.302440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.301827 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-xks-ca" pod "c0fe9fdc-4b19-494e-8c29-455263ac88f7" (UID: "c0fe9fdc-4b19-494e-8c29-455263ac88f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:12.305458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.303655 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fe9fdc-4b19-494e-8c29-455263ac88f7-kube-api-access-nmrb5" pod "c0fe9fdc-4b19-494e-8c29-455263ac88f7" (UID: "c0fe9fdc-4b19-494e-8c29-455263ac88f7"). InnerVolumeSpecName "kube-api-access-nmrb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:12.320800 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.320773 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:12.376441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.374749 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 30 04:19:12.376441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.374809 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" podUID="bc1b3d8c-6a80-400c-aebe-bbab2430d494" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 30 04:19:12.396576 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.396436 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.396576 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.396466 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0fe9fdc-4b19-494e-8c29-455263ac88f7-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.396576 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.396482 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmrb5\" (UniqueName: \"kubernetes.io/projected/c0fe9fdc-4b19-494e-8c29-455263ac88f7-kube-api-access-nmrb5\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:12.917447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.917189 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" Oct 30 04:19:12.917447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.917206 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq" event={"ID":"c0fe9fdc-4b19-494e-8c29-455263ac88f7","Type":"ContainerDied","Data":"53bcc8eef2f08deed3aaaf0662a755b7fa76fdc73fdf781ad9d2ba47ec1fef24"} Oct 30 04:19:12.917447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:12.917259 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53bcc8eef2f08deed3aaaf0662a755b7fa76fdc73fdf781ad9d2ba47ec1fef24" Oct 30 04:19:13.509897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:13.506990 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:13.509897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:13.508355 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:13.509897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:13.509626 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:19:14.221768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:14.221735 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:15.949394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:15.949358 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:17.636608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:17.636574 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:19.353339 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:19.353304 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:20.958975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:20.958932 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz"] Oct 30 04:19:20.959501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:20.959460 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c0fe9fdc-4b19-494e-8c29-455263ac88f7" containerName="revoke-keys" Oct 30 04:19:20.959501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:20.959473 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fe9fdc-4b19-494e-8c29-455263ac88f7" containerName="revoke-keys" Oct 30 04:19:20.959632 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:20.959617 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fe9fdc-4b19-494e-8c29-455263ac88f7" containerName="revoke-keys" Oct 30 04:19:20.964199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:20.964178 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:20.986230 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:20.986204 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz"] Oct 30 04:19:20.995322 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:20.995294 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:19:20.995621 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:20.995604 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:19:20.995701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:20.995644 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:19:20.996687 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:20.996670 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:19:21.009512 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.009485 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sv4f\" (UniqueName: \"kubernetes.io/projected/a8be392d-2a0b-4813-837a-0492164756fe-kube-api-access-2sv4f\") pod \"maas-api-revoke-sub-7090644b-f4qsz\" (UID: \"a8be392d-2a0b-4813-837a-0492164756fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:21.009648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.009630 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7090644b-f4qsz\" (UID: \"a8be392d-2a0b-4813-837a-0492164756fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:21.009694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.009669 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7090644b-f4qsz\" (UID: \"a8be392d-2a0b-4813-837a-0492164756fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:21.020561 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.020536 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:21.111296 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.111255 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7090644b-f4qsz\" (UID: \"a8be392d-2a0b-4813-837a-0492164756fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:21.111296 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.111296 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7090644b-f4qsz\" (UID: \"a8be392d-2a0b-4813-837a-0492164756fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:21.111614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.111401 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sv4f\" (UniqueName: \"kubernetes.io/projected/a8be392d-2a0b-4813-837a-0492164756fe-kube-api-access-2sv4f\") pod \"maas-api-revoke-sub-7090644b-f4qsz\" (UID: \"a8be392d-2a0b-4813-837a-0492164756fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:21.112088 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.112057 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7090644b-f4qsz\" (UID: \"a8be392d-2a0b-4813-837a-0492164756fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:21.113745 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.113722 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7090644b-f4qsz\" (UID: \"a8be392d-2a0b-4813-837a-0492164756fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:21.135358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.135333 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sv4f\" (UniqueName: \"kubernetes.io/projected/a8be392d-2a0b-4813-837a-0492164756fe-kube-api-access-2sv4f\") pod \"maas-api-revoke-sub-7090644b-f4qsz\" (UID: \"a8be392d-2a0b-4813-837a-0492164756fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:21.280821 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.280740 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:21.580543 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.580517 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz"] Oct 30 04:19:21.581714 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:19:21.581689 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8be392d_2a0b_4813_837a_0492164756fe.slice/crio-197873f5d6b43642a0ccb0136583e77f6095b60b7979bac0490cb4fc898c3e8a WatchSource:0}: Error finding container 197873f5d6b43642a0ccb0136583e77f6095b60b7979bac0490cb4fc898c3e8a: Status 404 returned error can't find the container with id 197873f5d6b43642a0ccb0136583e77f6095b60b7979bac0490cb4fc898c3e8a Oct 30 04:19:21.979908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.979876 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" event={"ID":"a8be392d-2a0b-4813-837a-0492164756fe","Type":"ContainerStarted","Data":"91b20cbe0da32f3ec8e2a8314ab288850ae91537f81692ef999ae90f231bf230"} Oct 30 04:19:21.980334 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:21.979930 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" event={"ID":"a8be392d-2a0b-4813-837a-0492164756fe","Type":"ContainerStarted","Data":"197873f5d6b43642a0ccb0136583e77f6095b60b7979bac0490cb4fc898c3e8a"} Oct 30 04:19:22.022520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:22.022467 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" podStartSLOduration=2.022452225 podStartE2EDuration="2.022452225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:22.020925385 +0000 UTC m=+1285.306800161" watchObservedRunningTime="2026-10-30 04:19:22.022452225 +0000 UTC m=+1285.308326999" Oct 30 04:19:22.727613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:22.727533 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:22.988693 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:22.988621 4793 generic.go:356] "Generic (PLEG): container finished" podID="a8be392d-2a0b-4813-837a-0492164756fe" containerID="91b20cbe0da32f3ec8e2a8314ab288850ae91537f81692ef999ae90f231bf230" exitCode=0 Oct 30 04:19:22.989071 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:22.988714 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" event={"ID":"a8be392d-2a0b-4813-837a-0492164756fe","Type":"ContainerDied","Data":"91b20cbe0da32f3ec8e2a8314ab288850ae91537f81692ef999ae90f231bf230"} Oct 30 04:19:24.142749 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.142725 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:24.218786 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.218739 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc"] Oct 30 04:19:24.221728 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.219560 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a8be392d-2a0b-4813-837a-0492164756fe" containerName="revoke-keys" Oct 30 04:19:24.221933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.221908 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8be392d-2a0b-4813-837a-0492164756fe" containerName="revoke-keys" Oct 30 04:19:24.222158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.222125 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8be392d-2a0b-4813-837a-0492164756fe" containerName="revoke-keys" Oct 30 04:19:24.230093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.230071 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc"] Oct 30 04:19:24.230400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.230333 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:24.243992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.243968 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-openshift-ca\") pod \"a8be392d-2a0b-4813-837a-0492164756fe\" (UID: \"a8be392d-2a0b-4813-837a-0492164756fe\") " Oct 30 04:19:24.244093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.244012 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-xks-ca\") pod \"a8be392d-2a0b-4813-837a-0492164756fe\" (UID: \"a8be392d-2a0b-4813-837a-0492164756fe\") " Oct 30 04:19:24.244093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.244065 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8be392d-2a0b-4813-837a-0492164756fe-kube-api-access-2sv4f\" (UniqueName: \"kubernetes.io/projected/a8be392d-2a0b-4813-837a-0492164756fe-kube-api-access-2sv4f\") pod \"a8be392d-2a0b-4813-837a-0492164756fe\" (UID: \"a8be392d-2a0b-4813-837a-0492164756fe\") " Oct 30 04:19:24.244413 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.244387 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-openshift-ca" pod "a8be392d-2a0b-4813-837a-0492164756fe" (UID: "a8be392d-2a0b-4813-837a-0492164756fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:24.244614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.244592 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:24.248662 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.248603 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8be392d-2a0b-4813-837a-0492164756fe-kube-api-access-2sv4f" pod "a8be392d-2a0b-4813-837a-0492164756fe" (UID: "a8be392d-2a0b-4813-837a-0492164756fe"). InnerVolumeSpecName "kube-api-access-2sv4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:24.248811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.248794 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-xks-ca" pod "a8be392d-2a0b-4813-837a-0492164756fe" (UID: "a8be392d-2a0b-4813-837a-0492164756fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:24.345104 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.345074 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb4jp\" (UniqueName: \"kubernetes.io/projected/eb3b71f0-6a23-4da7-b457-f53a13fb105e-kube-api-access-bb4jp\") pod \"maas-api-revoke-sub-9a7cd0f9-kcgpc\" (UID: \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:24.345289 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.345117 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a7cd0f9-kcgpc\" (UID: \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:24.345289 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.345139 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a7cd0f9-kcgpc\" (UID: \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:24.345474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.345452 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8be392d-2a0b-4813-837a-0492164756fe-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:24.345537 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.345481 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sv4f\" (UniqueName: \"kubernetes.io/projected/a8be392d-2a0b-4813-837a-0492164756fe-kube-api-access-2sv4f\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:24.447123 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.447090 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb4jp\" (UniqueName: \"kubernetes.io/projected/eb3b71f0-6a23-4da7-b457-f53a13fb105e-kube-api-access-bb4jp\") pod \"maas-api-revoke-sub-9a7cd0f9-kcgpc\" (UID: \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:24.447285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.447140 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a7cd0f9-kcgpc\" (UID: \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:24.447285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.447165 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a7cd0f9-kcgpc\" (UID: \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:24.447901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.447878 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a7cd0f9-kcgpc\" (UID: \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:24.449458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.449408 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a7cd0f9-kcgpc\" (UID: \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:24.470783 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.470755 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb4jp\" (UniqueName: \"kubernetes.io/projected/eb3b71f0-6a23-4da7-b457-f53a13fb105e-kube-api-access-bb4jp\") pod \"maas-api-revoke-sub-9a7cd0f9-kcgpc\" (UID: \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:24.551499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.540382 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:24.551499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.551146 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:24.570388 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.570358 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6"] Oct 30 04:19:24.575640 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.575617 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:24.589496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.589465 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6"] Oct 30 04:19:24.762051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.762015 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a214362-xvpr6\" (UID: \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:24.762249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.762072 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a214362-xvpr6\" (UID: \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:24.762249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.762172 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xq84\" (UniqueName: \"kubernetes.io/projected/09544a9a-3793-4039-aa9a-821a4e1aeaa0-kube-api-access-7xq84\") pod \"maas-api-revoke-sub-0a214362-xvpr6\" (UID: \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:24.863046 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.863010 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a214362-xvpr6\" (UID: \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:24.863205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.863085 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xq84\" (UniqueName: \"kubernetes.io/projected/09544a9a-3793-4039-aa9a-821a4e1aeaa0-kube-api-access-7xq84\") pod \"maas-api-revoke-sub-0a214362-xvpr6\" (UID: \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:24.863298 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.863273 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a214362-xvpr6\" (UID: \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:24.863822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.863788 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a214362-xvpr6\" (UID: \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:24.865684 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.865654 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a214362-xvpr6\" (UID: \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:24.890395 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.890362 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xq84\" (UniqueName: \"kubernetes.io/projected/09544a9a-3793-4039-aa9a-821a4e1aeaa0-kube-api-access-7xq84\") pod \"maas-api-revoke-sub-0a214362-xvpr6\" (UID: \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:24.897633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.897609 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:24.975666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:24.973851 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc"] Oct 30 04:19:25.008744 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:25.008675 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" Oct 30 04:19:25.009104 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:25.008986 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz" event={"ID":"a8be392d-2a0b-4813-837a-0492164756fe","Type":"ContainerDied","Data":"197873f5d6b43642a0ccb0136583e77f6095b60b7979bac0490cb4fc898c3e8a"} Oct 30 04:19:25.009104 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:25.009039 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="197873f5d6b43642a0ccb0136583e77f6095b60b7979bac0490cb4fc898c3e8a" Oct 30 04:19:25.024602 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:25.024410 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" event={"ID":"eb3b71f0-6a23-4da7-b457-f53a13fb105e","Type":"ContainerStarted","Data":"4176d02264067058ff8547ff74c8fc96df5026674c3c5ae62ca6da81e7a4f288"} Oct 30 04:19:25.297270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:25.294917 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6"] Oct 30 04:19:26.031824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:26.031788 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" event={"ID":"09544a9a-3793-4039-aa9a-821a4e1aeaa0","Type":"ContainerStarted","Data":"3d985fdec9a9229f7e486fa0b2bee2e76b991c39d40666189fcf6fe23f8464bf"} Oct 30 04:19:26.031983 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:26.031846 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" event={"ID":"09544a9a-3793-4039-aa9a-821a4e1aeaa0","Type":"ContainerStarted","Data":"fb814194511b27244195d778799dc2c044069a7ee65727a50cda5c1af33d73fe"} Oct 30 04:19:26.033295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:26.033265 4793 generic.go:356] "Generic (PLEG): container finished" podID="eb3b71f0-6a23-4da7-b457-f53a13fb105e" containerID="82d1d02ed13ce249ef6cd3130dcbb1c0da5106c37ef5a742cd68c8f410bc6e95" exitCode=0 Oct 30 04:19:26.033450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:26.033347 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" event={"ID":"eb3b71f0-6a23-4da7-b457-f53a13fb105e","Type":"ContainerDied","Data":"82d1d02ed13ce249ef6cd3130dcbb1c0da5106c37ef5a742cd68c8f410bc6e95"} Oct 30 04:19:26.074329 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:26.074275 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" podStartSLOduration=2.074262339 podStartE2EDuration="2.074262339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:26.073120753 +0000 UTC m=+1289.358995529" watchObservedRunningTime="2026-10-30 04:19:26.074262339 +0000 UTC m=+1289.360137113" Oct 30 04:19:26.380243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:26.380194 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:27.041964 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:27.041934 4793 generic.go:356] "Generic (PLEG): container finished" podID="09544a9a-3793-4039-aa9a-821a4e1aeaa0" containerID="3d985fdec9a9229f7e486fa0b2bee2e76b991c39d40666189fcf6fe23f8464bf" exitCode=0 Oct 30 04:19:27.042114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:27.042008 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" event={"ID":"09544a9a-3793-4039-aa9a-821a4e1aeaa0","Type":"ContainerDied","Data":"3d985fdec9a9229f7e486fa0b2bee2e76b991c39d40666189fcf6fe23f8464bf"} Oct 30 04:19:27.207795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:27.207773 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:27.216008 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:27.215985 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-xks-ca\") pod \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\" (UID: \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\") " Oct 30 04:19:27.216112 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:27.216097 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb3b71f0-6a23-4da7-b457-f53a13fb105e-kube-api-access-bb4jp\" (UniqueName: \"kubernetes.io/projected/eb3b71f0-6a23-4da7-b457-f53a13fb105e-kube-api-access-bb4jp\") pod \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\" (UID: \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\") " Oct 30 04:19:27.216175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:27.216160 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-openshift-ca\") pod \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\" (UID: \"eb3b71f0-6a23-4da7-b457-f53a13fb105e\") " Oct 30 04:19:27.216468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:27.216443 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-openshift-ca" pod "eb3b71f0-6a23-4da7-b457-f53a13fb105e" (UID: "eb3b71f0-6a23-4da7-b457-f53a13fb105e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:27.216676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:27.216660 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:27.217961 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:27.217939 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-xks-ca" pod "eb3b71f0-6a23-4da7-b457-f53a13fb105e" (UID: "eb3b71f0-6a23-4da7-b457-f53a13fb105e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:27.218095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:27.218076 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3b71f0-6a23-4da7-b457-f53a13fb105e-kube-api-access-bb4jp" pod "eb3b71f0-6a23-4da7-b457-f53a13fb105e" (UID: "eb3b71f0-6a23-4da7-b457-f53a13fb105e"). InnerVolumeSpecName "kube-api-access-bb4jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:27.317658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:27.317574 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb3b71f0-6a23-4da7-b457-f53a13fb105e-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:27.317658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:27.317611 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb4jp\" (UniqueName: \"kubernetes.io/projected/eb3b71f0-6a23-4da7-b457-f53a13fb105e-kube-api-access-bb4jp\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:28.050207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.050175 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" Oct 30 04:19:28.050207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.050174 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc" event={"ID":"eb3b71f0-6a23-4da7-b457-f53a13fb105e","Type":"ContainerDied","Data":"4176d02264067058ff8547ff74c8fc96df5026674c3c5ae62ca6da81e7a4f288"} Oct 30 04:19:28.050659 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.050250 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4176d02264067058ff8547ff74c8fc96df5026674c3c5ae62ca6da81e7a4f288" Oct 30 04:19:28.137580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.137396 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:28.230975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.230948 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:28.328558 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.328480 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-openshift-ca\") pod \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\" (UID: \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\") " Oct 30 04:19:28.328701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.328576 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09544a9a-3793-4039-aa9a-821a4e1aeaa0-kube-api-access-7xq84\" (UniqueName: \"kubernetes.io/projected/09544a9a-3793-4039-aa9a-821a4e1aeaa0-kube-api-access-7xq84\") pod \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\" (UID: \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\") " Oct 30 04:19:28.328701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.328664 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-xks-ca\") pod \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\" (UID: \"09544a9a-3793-4039-aa9a-821a4e1aeaa0\") " Oct 30 04:19:28.328888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.328860 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-openshift-ca" pod "09544a9a-3793-4039-aa9a-821a4e1aeaa0" (UID: "09544a9a-3793-4039-aa9a-821a4e1aeaa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:28.329107 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.329081 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:28.330636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.330613 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09544a9a-3793-4039-aa9a-821a4e1aeaa0-kube-api-access-7xq84" pod "09544a9a-3793-4039-aa9a-821a4e1aeaa0" (UID: "09544a9a-3793-4039-aa9a-821a4e1aeaa0"). InnerVolumeSpecName "kube-api-access-7xq84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:28.330779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.330766 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-xks-ca" pod "09544a9a-3793-4039-aa9a-821a4e1aeaa0" (UID: "09544a9a-3793-4039-aa9a-821a4e1aeaa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:28.430495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.430465 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xq84\" (UniqueName: \"kubernetes.io/projected/09544a9a-3793-4039-aa9a-821a4e1aeaa0-kube-api-access-7xq84\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:28.430495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:28.430490 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09544a9a-3793-4039-aa9a-821a4e1aeaa0-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:29.057242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:29.057217 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" Oct 30 04:19:29.057649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:29.057251 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6" event={"ID":"09544a9a-3793-4039-aa9a-821a4e1aeaa0","Type":"ContainerDied","Data":"fb814194511b27244195d778799dc2c044069a7ee65727a50cda5c1af33d73fe"} Oct 30 04:19:29.057649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:29.057299 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb814194511b27244195d778799dc2c044069a7ee65727a50cda5c1af33d73fe" Oct 30 04:19:29.938247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:29.938207 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:31.625107 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:31.625070 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:33.218779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:33.218731 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:35.011807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:35.011765 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:36.831598 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:36.831564 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:38.610646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:38.610604 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:40.011487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.011244 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8"] Oct 30 04:19:40.012736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.012699 4793 cpu_manager.go:414] "containerMap: removing container" podUID="09544a9a-3793-4039-aa9a-821a4e1aeaa0" containerName="revoke-keys" Oct 30 04:19:40.012736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.012729 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09544a9a-3793-4039-aa9a-821a4e1aeaa0" containerName="revoke-keys" Oct 30 04:19:40.012917 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.012797 4793 cpu_manager.go:414] "containerMap: removing container" podUID="eb3b71f0-6a23-4da7-b457-f53a13fb105e" containerName="revoke-keys" Oct 30 04:19:40.012917 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.012804 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb3b71f0-6a23-4da7-b457-f53a13fb105e" containerName="revoke-keys" Oct 30 04:19:40.013000 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.012927 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb3b71f0-6a23-4da7-b457-f53a13fb105e" containerName="revoke-keys" Oct 30 04:19:40.013000 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.012939 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="09544a9a-3793-4039-aa9a-821a4e1aeaa0" containerName="revoke-keys" Oct 30 04:19:40.036285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.018072 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.036285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.029267 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8"] Oct 30 04:19:40.038870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.038163 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"default-dockercfg-zcchk\"" Oct 30 04:19:40.038870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.038787 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"e2e-conflict-a-54a3def2-kserve-self-signed-certs\"" Oct 30 04:19:40.047616 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.047593 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"openshift-service-ca.crt\"" Oct 30 04:19:40.047710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.047648 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6796b1\"/\"kube-root-ca.crt\"" Oct 30 04:19:40.067291 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.067264 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b193949-4e87-4992-8247-61849a043981-tls-certs\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.067406 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.067382 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-tmp-dir\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.067485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.067474 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-home\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.067585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.067562 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-model-cache\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.067641 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.067622 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl4xf\" (UniqueName: \"kubernetes.io/projected/5b193949-4e87-4992-8247-61849a043981-kube-api-access-sl4xf\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.067753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.067737 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-dshm\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.168538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.168512 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-tmp-dir\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.168688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.168578 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-home\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.168688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.168601 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-model-cache\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.168688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.168632 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl4xf\" (UniqueName: \"kubernetes.io/projected/5b193949-4e87-4992-8247-61849a043981-kube-api-access-sl4xf\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.168688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.168674 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-dshm\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.168902 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.168718 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b193949-4e87-4992-8247-61849a043981-tls-certs\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.168956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.168907 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-tmp-dir\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.169060 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.169035 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-model-cache\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.169241 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.169219 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-home\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.169651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.169628 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:40.171619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.171592 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-dshm\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.171801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.171783 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b193949-4e87-4992-8247-61849a043981-tls-certs\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.192481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.192454 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl4xf\" (UniqueName: \"kubernetes.io/projected/5b193949-4e87-4992-8247-61849a043981-kube-api-access-sl4xf\") pod \"e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.351882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.351781 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:40.506081 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.506048 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:40.670490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:40.666985 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8"] Oct 30 04:19:40.676846 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:19:40.676818 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b193949_4e87_4992_8247_61849a043981.slice/crio-7d982cbb7d3eaab515fd895414c2d73353abd3ce2e7100ae9a3173a0638dac9a WatchSource:0}: Error finding container 7d982cbb7d3eaab515fd895414c2d73353abd3ce2e7100ae9a3173a0638dac9a: Status 404 returned error can't find the container with id 7d982cbb7d3eaab515fd895414c2d73353abd3ce2e7100ae9a3173a0638dac9a Oct 30 04:19:41.148013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:41.147976 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" event={"ID":"5b193949-4e87-4992-8247-61849a043981","Type":"ContainerStarted","Data":"24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669"} Oct 30 04:19:41.148459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:41.148023 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" event={"ID":"5b193949-4e87-4992-8247-61849a043981","Type":"ContainerStarted","Data":"7d982cbb7d3eaab515fd895414c2d73353abd3ce2e7100ae9a3173a0638dac9a"} Oct 30 04:19:41.192243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:41.192186 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" podStartSLOduration=2.192172596 podStartE2EDuration="2.192172596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:41.190595553 +0000 UTC m=+1304.476470327" watchObservedRunningTime="2026-10-30 04:19:41.192172596 +0000 UTC m=+1304.478047372" Oct 30 04:19:41.218528 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:19:41.218494 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:19:41.351870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:41.351839 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:42.090623 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.090583 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch"] Oct 30 04:19:42.094651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.094626 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:42.113076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.113053 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch"] Oct 30 04:19:42.121948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.121923 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:19:42.122042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.121931 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:19:42.122042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.121968 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:19:42.123795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.123777 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:19:42.203994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.203960 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8572a84-zjxch\" (UID: \"417dfae5-0ce8-414c-b0d4-86321d544654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:42.204474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.204004 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8572a84-zjxch\" (UID: \"417dfae5-0ce8-414c-b0d4-86321d544654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:42.204474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.204263 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv4wp\" (UniqueName: \"kubernetes.io/projected/417dfae5-0ce8-414c-b0d4-86321d544654-kube-api-access-cv4wp\") pod \"maas-api-revoke-sub-e8572a84-zjxch\" (UID: \"417dfae5-0ce8-414c-b0d4-86321d544654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:42.305482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.305448 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv4wp\" (UniqueName: \"kubernetes.io/projected/417dfae5-0ce8-414c-b0d4-86321d544654-kube-api-access-cv4wp\") pod \"maas-api-revoke-sub-e8572a84-zjxch\" (UID: \"417dfae5-0ce8-414c-b0d4-86321d544654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:42.305645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.305505 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8572a84-zjxch\" (UID: \"417dfae5-0ce8-414c-b0d4-86321d544654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:42.305645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.305527 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8572a84-zjxch\" (UID: \"417dfae5-0ce8-414c-b0d4-86321d544654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:42.306218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.306198 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8572a84-zjxch\" (UID: \"417dfae5-0ce8-414c-b0d4-86321d544654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:42.307956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.307933 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8572a84-zjxch\" (UID: \"417dfae5-0ce8-414c-b0d4-86321d544654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:42.328947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.328917 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv4wp\" (UniqueName: \"kubernetes.io/projected/417dfae5-0ce8-414c-b0d4-86321d544654-kube-api-access-cv4wp\") pod \"maas-api-revoke-sub-e8572a84-zjxch\" (UID: \"417dfae5-0ce8-414c-b0d4-86321d544654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:42.395610 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.395520 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:42.410569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.410542 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:42.716738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:42.716709 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch"] Oct 30 04:19:42.717122 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:19:42.717100 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417dfae5_0ce8_414c_b0d4_86321d544654.slice/crio-e8ca4b07d10463a00322de3cf0422f894208fcd4a05624d52adab7f4df85105f WatchSource:0}: Error finding container e8ca4b07d10463a00322de3cf0422f894208fcd4a05624d52adab7f4df85105f: Status 404 returned error can't find the container with id e8ca4b07d10463a00322de3cf0422f894208fcd4a05624d52adab7f4df85105f Oct 30 04:19:43.163993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:43.163955 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" event={"ID":"417dfae5-0ce8-414c-b0d4-86321d544654","Type":"ContainerStarted","Data":"bb3005d721765549bbb8a72fe67f284e8d59615df6c82f996da6cef4c0ab1bc8"} Oct 30 04:19:43.164186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:43.164014 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" event={"ID":"417dfae5-0ce8-414c-b0d4-86321d544654","Type":"ContainerStarted","Data":"e8ca4b07d10463a00322de3cf0422f894208fcd4a05624d52adab7f4df85105f"} Oct 30 04:19:43.251730 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:43.251627 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" podStartSLOduration=1.25161438 podStartE2EDuration="1.25161438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:43.249603833 +0000 UTC m=+1306.535478608" watchObservedRunningTime="2026-10-30 04:19:43.25161438 +0000 UTC m=+1306.537489154" Oct 30 04:19:44.179090 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:44.179054 4793 generic.go:356] "Generic (PLEG): container finished" podID="417dfae5-0ce8-414c-b0d4-86321d544654" containerID="bb3005d721765549bbb8a72fe67f284e8d59615df6c82f996da6cef4c0ab1bc8" exitCode=0 Oct 30 04:19:44.179283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:44.179113 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" event={"ID":"417dfae5-0ce8-414c-b0d4-86321d544654","Type":"ContainerDied","Data":"bb3005d721765549bbb8a72fe67f284e8d59615df6c82f996da6cef4c0ab1bc8"} Oct 30 04:19:44.268740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:44.268695 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:45.330839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:45.330817 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:45.441927 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:45.441846 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417dfae5-0ce8-414c-b0d4-86321d544654-kube-api-access-cv4wp\" (UniqueName: \"kubernetes.io/projected/417dfae5-0ce8-414c-b0d4-86321d544654-kube-api-access-cv4wp\") pod \"417dfae5-0ce8-414c-b0d4-86321d544654\" (UID: \"417dfae5-0ce8-414c-b0d4-86321d544654\") " Oct 30 04:19:45.441927 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:45.441903 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-openshift-ca\") pod \"417dfae5-0ce8-414c-b0d4-86321d544654\" (UID: \"417dfae5-0ce8-414c-b0d4-86321d544654\") " Oct 30 04:19:45.442140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:45.441984 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-xks-ca\") pod \"417dfae5-0ce8-414c-b0d4-86321d544654\" (UID: \"417dfae5-0ce8-414c-b0d4-86321d544654\") " Oct 30 04:19:45.442291 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:45.442264 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-openshift-ca" pod "417dfae5-0ce8-414c-b0d4-86321d544654" (UID: "417dfae5-0ce8-414c-b0d4-86321d544654"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:45.442486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:45.442469 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:45.444002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:45.443977 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-xks-ca" pod "417dfae5-0ce8-414c-b0d4-86321d544654" (UID: "417dfae5-0ce8-414c-b0d4-86321d544654"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:45.444085 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:45.443984 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417dfae5-0ce8-414c-b0d4-86321d544654-kube-api-access-cv4wp" pod "417dfae5-0ce8-414c-b0d4-86321d544654" (UID: "417dfae5-0ce8-414c-b0d4-86321d544654"). InnerVolumeSpecName "kube-api-access-cv4wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:45.543969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:45.543927 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417dfae5-0ce8-414c-b0d4-86321d544654-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:45.543969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:45.543965 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv4wp\" (UniqueName: \"kubernetes.io/projected/417dfae5-0ce8-414c-b0d4-86321d544654-kube-api-access-cv4wp\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:46.144470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:46.144431 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:46.192762 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:46.192737 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" Oct 30 04:19:46.192895 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:46.192772 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch" event={"ID":"417dfae5-0ce8-414c-b0d4-86321d544654","Type":"ContainerDied","Data":"e8ca4b07d10463a00322de3cf0422f894208fcd4a05624d52adab7f4df85105f"} Oct 30 04:19:46.192895 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:46.192830 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8ca4b07d10463a00322de3cf0422f894208fcd4a05624d52adab7f4df85105f" Oct 30 04:19:47.973233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:47.973188 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:49.704870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:49.704829 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:50.352768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:50.352732 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:50.366153 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:50.366124 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:51.185182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.185151 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822"] Oct 30 04:19:51.185728 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.185713 4793 cpu_manager.go:414] "containerMap: removing container" podUID="417dfae5-0ce8-414c-b0d4-86321d544654" containerName="revoke-keys" Oct 30 04:19:51.185772 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.185730 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417dfae5-0ce8-414c-b0d4-86321d544654" containerName="revoke-keys" Oct 30 04:19:51.185850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.185840 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="417dfae5-0ce8-414c-b0d4-86321d544654" containerName="revoke-keys" Oct 30 04:19:51.189308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.189285 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.222638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.222612 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822"] Oct 30 04:19:51.246455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.246346 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:19:51.318106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.318070 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.318266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.318123 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.318266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.318155 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f67zh\" (UniqueName: \"kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-kube-api-access-f67zh\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.318266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.318193 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.318266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.318218 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.318266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.318254 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.318570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.318276 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.318570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.318300 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.318570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.318357 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a607605-2415-4bd8-a313-47cd6089c8a2-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.419734 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.419691 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a607605-2415-4bd8-a313-47cd6089c8a2-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.421148 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.420804 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a607605-2415-4bd8-a313-47cd6089c8a2-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.421461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.421436 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.422482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.422433 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.422619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.422603 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.422766 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.422749 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f67zh\" (UniqueName: \"kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-kube-api-access-f67zh\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.423356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.423335 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.424240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.424222 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.424445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.424411 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.424844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.424815 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.424970 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.424876 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.425443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.425379 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.425767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.424162 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.426412 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.426387 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.430632 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.430605 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.432186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.432161 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.461488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.461411 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f67zh\" (UniqueName: \"kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-kube-api-access-f67zh\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.462515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.462496 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.506225 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.506204 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:51.537532 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.537299 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:51.835843 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:19:51.835810 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a607605_2415_4bd8_a313_47cd6089c8a2.slice/crio-81e7529577319a8528b4ca423e2b24d8273e3f9b023b397c0b312fd787de991e WatchSource:0}: Error finding container 81e7529577319a8528b4ca423e2b24d8273e3f9b023b397c0b312fd787de991e: Status 404 returned error can't find the container with id 81e7529577319a8528b4ca423e2b24d8273e3f9b023b397c0b312fd787de991e Oct 30 04:19:51.836113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.836094 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822"] Oct 30 04:19:51.838499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.838470 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:19:51.838611 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.838531 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:19:51.838611 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:51.838558 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:19:52.239519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.239471 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" event={"ID":"7a607605-2415-4bd8-a313-47cd6089c8a2","Type":"ContainerStarted","Data":"d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a"} Oct 30 04:19:52.240019 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.239543 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" event={"ID":"7a607605-2415-4bd8-a313-47cd6089c8a2","Type":"ContainerStarted","Data":"81e7529577319a8528b4ca423e2b24d8273e3f9b023b397c0b312fd787de991e"} Oct 30 04:19:52.288709 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.288660 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" podStartSLOduration=1.288646829 podStartE2EDuration="1.288646829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:19:52.285284061 +0000 UTC m=+1315.571158860" watchObservedRunningTime="2026-10-30 04:19:52.288646829 +0000 UTC m=+1315.574521603" Oct 30 04:19:52.390115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.390084 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr"] Oct 30 04:19:52.394233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.394213 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:52.412049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.412022 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr"] Oct 30 04:19:52.412243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.412224 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:19:52.412307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.412259 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:19:52.412307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.412270 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:19:52.412398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.412348 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:19:52.436453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.429152 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:52.507366 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.507275 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:52.508742 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.508715 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 04:19:52.508850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.508774 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" podUID="7a607605-2415-4bd8-a313-47cd6089c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 04:19:52.539400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.539372 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76dede6a-6jbxr\" (UID: \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:52.539555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.539531 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76dede6a-6jbxr\" (UID: \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:52.539634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.539619 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjhmn\" (UniqueName: \"kubernetes.io/projected/da2b78f6-613a-4163-8ba0-fcfbd227d95a-kube-api-access-rjhmn\") pod \"maas-api-revoke-sub-76dede6a-6jbxr\" (UID: \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:52.640604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.640573 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76dede6a-6jbxr\" (UID: \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:52.640770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.640630 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjhmn\" (UniqueName: \"kubernetes.io/projected/da2b78f6-613a-4163-8ba0-fcfbd227d95a-kube-api-access-rjhmn\") pod \"maas-api-revoke-sub-76dede6a-6jbxr\" (UID: \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:52.640770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.640704 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76dede6a-6jbxr\" (UID: \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:52.641377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.641353 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76dede6a-6jbxr\" (UID: \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:52.642946 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.642920 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76dede6a-6jbxr\" (UID: \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:52.671440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.669544 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjhmn\" (UniqueName: \"kubernetes.io/projected/da2b78f6-613a-4163-8ba0-fcfbd227d95a-kube-api-access-rjhmn\") pod \"maas-api-revoke-sub-76dede6a-6jbxr\" (UID: \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:52.708151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:52.708121 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:53.034843 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:53.034797 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr"] Oct 30 04:19:53.035623 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:19:53.035597 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2b78f6_613a_4163_8ba0_fcfbd227d95a.slice/crio-1c09ee16931a679eb88a71638e36d712f1dfcff4bcc91b86313cea086380746b WatchSource:0}: Error finding container 1c09ee16931a679eb88a71638e36d712f1dfcff4bcc91b86313cea086380746b: Status 404 returned error can't find the container with id 1c09ee16931a679eb88a71638e36d712f1dfcff4bcc91b86313cea086380746b Oct 30 04:19:53.126819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:53.126782 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:53.249175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:53.249139 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" event={"ID":"da2b78f6-613a-4163-8ba0-fcfbd227d95a","Type":"ContainerStarted","Data":"1c09ee16931a679eb88a71638e36d712f1dfcff4bcc91b86313cea086380746b"} Oct 30 04:19:53.506689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:53.506657 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 04:19:53.506817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:53.506706 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" podUID="7a607605-2415-4bd8-a313-47cd6089c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 04:19:54.271441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:54.266680 4793 generic.go:356] "Generic (PLEG): container finished" podID="da2b78f6-613a-4163-8ba0-fcfbd227d95a" containerID="fa060d7faf23e84b114891ae64743bb87c3ddd3de57de7741baa65423b865190" exitCode=0 Oct 30 04:19:54.271441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:54.267835 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" event={"ID":"da2b78f6-613a-4163-8ba0-fcfbd227d95a","Type":"ContainerDied","Data":"fa060d7faf23e84b114891ae64743bb87c3ddd3de57de7741baa65423b865190"} Oct 30 04:19:54.506662 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:54.506574 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 04:19:54.506662 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:54.506642 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" podUID="7a607605-2415-4bd8-a313-47cd6089c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 04:19:54.942444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:54.942375 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:55.481629 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:55.481594 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:55.506818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:55.506792 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 04:19:55.506945 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:55.506856 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" podUID="7a607605-2415-4bd8-a313-47cd6089c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 04:19:55.604262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:55.603298 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-xks-ca\") pod \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\" (UID: \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\") " Oct 30 04:19:55.604262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:55.603361 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da2b78f6-613a-4163-8ba0-fcfbd227d95a-kube-api-access-rjhmn\" (UniqueName: \"kubernetes.io/projected/da2b78f6-613a-4163-8ba0-fcfbd227d95a-kube-api-access-rjhmn\") pod \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\" (UID: \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\") " Oct 30 04:19:55.604262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:55.603453 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-openshift-ca\") pod \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\" (UID: \"da2b78f6-613a-4163-8ba0-fcfbd227d95a\") " Oct 30 04:19:55.604963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:55.604652 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-openshift-ca" pod "da2b78f6-613a-4163-8ba0-fcfbd227d95a" (UID: "da2b78f6-613a-4163-8ba0-fcfbd227d95a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:19:55.606880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:55.606848 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2b78f6-613a-4163-8ba0-fcfbd227d95a-kube-api-access-rjhmn" pod "da2b78f6-613a-4163-8ba0-fcfbd227d95a" (UID: "da2b78f6-613a-4163-8ba0-fcfbd227d95a"). InnerVolumeSpecName "kube-api-access-rjhmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:19:55.607003 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:55.606864 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-xks-ca" pod "da2b78f6-613a-4163-8ba0-fcfbd227d95a" (UID: "da2b78f6-613a-4163-8ba0-fcfbd227d95a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:19:55.705451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:55.704779 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:55.705451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:55.704813 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjhmn\" (UniqueName: \"kubernetes.io/projected/da2b78f6-613a-4163-8ba0-fcfbd227d95a-kube-api-access-rjhmn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:55.705451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:55.704828 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da2b78f6-613a-4163-8ba0-fcfbd227d95a-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:19:56.289002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:56.288963 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" Oct 30 04:19:56.289002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:56.288990 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr" event={"ID":"da2b78f6-613a-4163-8ba0-fcfbd227d95a","Type":"ContainerDied","Data":"1c09ee16931a679eb88a71638e36d712f1dfcff4bcc91b86313cea086380746b"} Oct 30 04:19:56.289244 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:56.289039 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c09ee16931a679eb88a71638e36d712f1dfcff4bcc91b86313cea086380746b" Oct 30 04:19:56.507267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:56.507225 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 04:19:56.507674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:56.507307 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" podUID="7a607605-2415-4bd8-a313-47cd6089c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 04:19:56.742298 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:56.742260 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:57.507695 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:57.507651 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 30 04:19:57.508379 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:57.508346 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" podUID="7a607605-2415-4bd8-a313-47cd6089c8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 30 04:19:58.442774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:58.442735 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:58.649764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:58.649724 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:19:59.233654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:59.233171 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:59.234620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:59.234595 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:19:59.253779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:19:59.253743 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:20:00.262237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:00.262203 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:01.917839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:01.917801 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:05.039828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.039791 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx"] Oct 30 04:20:05.040521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.040500 4793 cpu_manager.go:414] "containerMap: removing container" podUID="da2b78f6-613a-4163-8ba0-fcfbd227d95a" containerName="revoke-keys" Oct 30 04:20:05.040616 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.040522 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da2b78f6-613a-4163-8ba0-fcfbd227d95a" containerName="revoke-keys" Oct 30 04:20:05.040681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.040668 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="da2b78f6-613a-4163-8ba0-fcfbd227d95a" containerName="revoke-keys" Oct 30 04:20:05.045299 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.045277 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.064514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.064482 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx"] Oct 30 04:20:05.070980 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.070953 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5d74b9\"/\"distinct-w2-5d74b9-kserve-self-signed-certs\"" Oct 30 04:20:05.134738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.134708 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-home\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.134892 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.134749 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-model-cache\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.134892 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.134772 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4g9r\" (UniqueName: \"kubernetes.io/projected/1715329f-b8ff-48a2-8c08-32e93a0b14da-kube-api-access-f4g9r\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.134995 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.134910 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-tmp-dir\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.134995 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.134972 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-dshm\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.135082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.135063 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1715329f-b8ff-48a2-8c08-32e93a0b14da-tls-certs\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.145620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.145591 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:05.245356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.245316 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-tmp-dir\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.245780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.245750 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-dshm\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.245963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.245945 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1715329f-b8ff-48a2-8c08-32e93a0b14da-tls-certs\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.259264 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.259232 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-home\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.259730 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.259563 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-model-cache\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.260014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.259990 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4g9r\" (UniqueName: \"kubernetes.io/projected/1715329f-b8ff-48a2-8c08-32e93a0b14da-kube-api-access-f4g9r\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.266475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.262089 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-home\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.266475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.245816 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-tmp-dir\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.266475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.262404 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-model-cache\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.270612 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.269227 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-dshm\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.278054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.278021 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1715329f-b8ff-48a2-8c08-32e93a0b14da-tls-certs\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.298500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.298395 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4g9r\" (UniqueName: \"kubernetes.io/projected/1715329f-b8ff-48a2-8c08-32e93a0b14da-kube-api-access-f4g9r\") pod \"distinct-w2-5d74b9-kserve-69b96f95bd-pltdx\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.348493 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.348461 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:05.366672 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.366379 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:05.551207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.551124 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:05.690098 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:20:05.690069 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1715329f_b8ff_48a2_8c08_32e93a0b14da.slice/crio-357871f650e10d4ec7c4322c0a0de3180841ba2d8a95901365237672fa1043a1 WatchSource:0}: Error finding container 357871f650e10d4ec7c4322c0a0de3180841ba2d8a95901365237672fa1043a1: Status 404 returned error can't find the container with id 357871f650e10d4ec7c4322c0a0de3180841ba2d8a95901365237672fa1043a1 Oct 30 04:20:05.692063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:05.692030 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx"] Oct 30 04:20:06.389656 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:06.389624 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" event={"ID":"1715329f-b8ff-48a2-8c08-32e93a0b14da","Type":"ContainerStarted","Data":"607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6"} Oct 30 04:20:06.390196 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:06.390163 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" event={"ID":"1715329f-b8ff-48a2-8c08-32e93a0b14da","Type":"ContainerStarted","Data":"357871f650e10d4ec7c4322c0a0de3180841ba2d8a95901365237672fa1043a1"} Oct 30 04:20:06.433571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:06.433522 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" podStartSLOduration=1.433509088 podStartE2EDuration="1.433509088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:06.431619241 +0000 UTC m=+1329.717494015" watchObservedRunningTime="2026-10-30 04:20:06.433509088 +0000 UTC m=+1329.719383864" Oct 30 04:20:07.105305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:07.105250 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:07.366992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:07.366916 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:08.815160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:08.815121 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:10.505270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:10.505230 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:10.866062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:10.865981 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:11.218966 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:20:11.218926 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:20:12.198463 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:12.198399 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:13.810896 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:13.810862 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:15.367291 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:15.367261 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:15.380130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:15.380105 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:15.478156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:15.478120 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 04:20:15.561186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:15.561149 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:17.349923 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:17.349874 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:19.042389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:19.042354 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:20.700374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:20.700338 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:21.416328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:21.416286 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:22.398635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:22.398598 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:24.101093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:24.101058 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:25.119560 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:25.119517 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:25.320716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:25.320672 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:25.710863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:25.710831 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:27.512470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:27.512426 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:29.311151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:29.311106 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:31.094300 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:31.094262 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:32.798522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:32.798483 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:33.956147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:33.956111 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:34.599249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:34.599211 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:35.409891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.409863 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8"] Oct 30 04:20:35.544489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.544400 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:35.833182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.833144 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j"] Oct 30 04:20:35.838196 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.838172 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:35.860676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.860643 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j"] Oct 30 04:20:35.939269 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.939239 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqfcb\" (UniqueName: \"kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-kube-api-access-xqfcb\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:35.939455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.939282 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:35.939455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.939303 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:35.939555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.939481 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:35.939555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.939533 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:35.939635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.939604 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:35.939678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.939659 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:35.939817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.939794 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:35.939885 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:35.939837 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.041209 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.041176 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.041356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.041257 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.041356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.041289 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.041479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.041358 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.041479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.041445 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.041593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.041531 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.041593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.041562 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.041688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.041619 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfcb\" (UniqueName: \"kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-kube-api-access-xqfcb\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.041688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.041666 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.042062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.042034 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.042204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.042064 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.042204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.042108 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.042348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.042212 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.042450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.042400 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.043602 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.043575 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.043925 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.043902 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.065082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.065061 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqfcb\" (UniqueName: \"kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-kube-api-access-xqfcb\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.065175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.065112 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.153603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.153540 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:36.162290 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.162253 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:36.371925 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.371884 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:36.476967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.476937 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j"] Oct 30 04:20:36.477382 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:20:36.476969 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a4742f_1003_46d5_bdad_bc7eebe769dc.slice/crio-de9140c4cd92b582bc741676179d35b82cf7647da5d533ded71955bad6c16cef WatchSource:0}: Error finding container de9140c4cd92b582bc741676179d35b82cf7647da5d533ded71955bad6c16cef: Status 404 returned error can't find the container with id de9140c4cd92b582bc741676179d35b82cf7647da5d533ded71955bad6c16cef Oct 30 04:20:36.479449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.479393 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:20:36.479588 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.479514 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:20:36.479588 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.479562 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:20:36.610953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.610845 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" event={"ID":"a5a4742f-1003-46d5-bdad-bc7eebe769dc","Type":"ContainerStarted","Data":"64dc6156ffd278687bfccbbc5125535aa63bcfbbf49eb70b9c0b8476cbf47b88"} Oct 30 04:20:36.610953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.610918 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" event={"ID":"a5a4742f-1003-46d5-bdad-bc7eebe769dc","Type":"ContainerStarted","Data":"de9140c4cd92b582bc741676179d35b82cf7647da5d533ded71955bad6c16cef"} Oct 30 04:20:36.658999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:36.658937 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" podStartSLOduration=1.658922537 podStartE2EDuration="1.658922537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:36.656184733 +0000 UTC m=+1359.942059509" watchObservedRunningTime="2026-10-30 04:20:36.658922537 +0000 UTC m=+1359.944797311" Oct 30 04:20:37.157442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:37.154239 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:37.175938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:37.175564 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 04:20:37.175938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:37.175641 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" podUID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 04:20:38.122913 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.122866 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:38.158591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.158552 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 04:20:38.158866 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.158824 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" podUID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 04:20:38.308134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.308088 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7"] Oct 30 04:20:38.312212 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.312191 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.376492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.373609 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7"] Oct 30 04:20:38.392689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.392654 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wnch\" (UniqueName: \"kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-kube-api-access-4wnch\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.392880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.392707 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.392880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.392737 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.392880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.392788 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.392880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.392811 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.392880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.392858 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.393158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.392888 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.393158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.392947 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a6fc57-013f-4b08-8768-f01a594f6c64-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.393158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.393046 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.493872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.493840 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wnch\" (UniqueName: \"kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-kube-api-access-4wnch\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.494087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.493882 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.494087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.493903 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.494087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.493949 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.494087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.493973 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.494087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.494022 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.494087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.494051 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.494446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.494125 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a6fc57-013f-4b08-8768-f01a594f6c64-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.494446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.494230 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.494446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.494366 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.494628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.494507 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.494723 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.494700 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.494781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.494707 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.495074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.495051 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a6fc57-013f-4b08-8768-f01a594f6c64-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.496900 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.496868 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.497383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.497356 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.520535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.520510 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.520634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.520547 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wnch\" (UniqueName: \"kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-kube-api-access-4wnch\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.627810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.627724 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:38.992567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.992522 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7"] Oct 30 04:20:38.993932 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:20:38.993908 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a6fc57_013f_4b08_8768_f01a594f6c64.slice/crio-6461a3aa843dd4234c3abab62fccb653c20cd25ba154696e33597db26b4f4e52 WatchSource:0}: Error finding container 6461a3aa843dd4234c3abab62fccb653c20cd25ba154696e33597db26b4f4e52: Status 404 returned error can't find the container with id 6461a3aa843dd4234c3abab62fccb653c20cd25ba154696e33597db26b4f4e52 Oct 30 04:20:38.997089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.997053 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:20:38.997191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.997135 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:20:38.997191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:38.997184 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:20:39.186018 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:39.185612 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 04:20:39.186018 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:39.185669 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" podUID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 04:20:39.292729 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:39.292091 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:39.633344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:39.633254 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" event={"ID":"62a6fc57-013f-4b08-8768-f01a594f6c64","Type":"ContainerStarted","Data":"31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7"} Oct 30 04:20:39.633344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:39.633315 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" event={"ID":"62a6fc57-013f-4b08-8768-f01a594f6c64","Type":"ContainerStarted","Data":"6461a3aa843dd4234c3abab62fccb653c20cd25ba154696e33597db26b4f4e52"} Oct 30 04:20:39.681443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:39.681099 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" podStartSLOduration=1.681081046 podStartE2EDuration="1.681081046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:39.67891122 +0000 UTC m=+1362.964786014" watchObservedRunningTime="2026-10-30 04:20:39.681081046 +0000 UTC m=+1362.966955822" Oct 30 04:20:40.073805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:40.068057 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:40.158441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:40.157366 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 04:20:40.158441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:40.157448 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" podUID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 04:20:40.628295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:40.628259 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:40.629466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:40.629443 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 04:20:40.629527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:40.629492 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" podUID="62a6fc57-013f-4b08-8768-f01a594f6c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 04:20:40.824550 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:40.824500 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:41.158451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:41.157761 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 04:20:41.158451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:41.157819 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" podUID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 04:20:41.238321 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:20:41.238289 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:20:41.628064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:41.628028 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 04:20:41.628253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:41.628077 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" podUID="62a6fc57-013f-4b08-8768-f01a594f6c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 04:20:41.981438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:41.981189 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:42.156413 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:42.156379 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 04:20:42.156715 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:42.156686 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" podUID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 04:20:42.628961 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:42.628919 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 04:20:42.629143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:42.628993 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" podUID="62a6fc57-013f-4b08-8768-f01a594f6c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 04:20:43.159441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:43.158112 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 04:20:43.159441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:43.158188 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" podUID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 04:20:43.628800 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:43.628759 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 04:20:43.628957 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:43.628827 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" podUID="62a6fc57-013f-4b08-8768-f01a594f6c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 04:20:44.004445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:44.003786 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:44.172534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:44.171683 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:44.173007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:44.172789 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:44.185703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:44.185673 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:20:44.628300 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:44.628263 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 04:20:44.628547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:44.628328 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" podUID="62a6fc57-013f-4b08-8768-f01a594f6c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 04:20:45.628578 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:45.628543 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 30 04:20:45.628959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:45.628597 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" podUID="62a6fc57-013f-4b08-8768-f01a594f6c64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 30 04:20:45.807218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:45.807116 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:47.362122 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:47.362092 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:47.362595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:47.362457 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:47.363309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:47.363286 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:20:47.497107 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:47.497071 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:49.090107 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:49.090065 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:50.431612 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.431569 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" podUID="5b193949-4e87-4992-8247-61849a043981" containerName="main" containerID="cri-o://24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669" gracePeriod=45 Oct 30 04:20:50.694696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.694612 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:20:50.743784 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.743750 4793 generic.go:356] "Generic (PLEG): container finished" podID="5b193949-4e87-4992-8247-61849a043981" containerID="24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669" exitCode=0 Oct 30 04:20:50.744149 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.744128 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" event={"ID":"5b193949-4e87-4992-8247-61849a043981","Type":"ContainerDied","Data":"24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669"} Oct 30 04:20:50.744265 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.744253 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" event={"ID":"5b193949-4e87-4992-8247-61849a043981","Type":"ContainerDied","Data":"7d982cbb7d3eaab515fd895414c2d73353abd3ce2e7100ae9a3173a0638dac9a"} Oct 30 04:20:50.744351 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.744341 4793 scope.go:122] "RemoveContainer" containerID="24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669" Oct 30 04:20:50.744603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.744584 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8" Oct 30 04:20:50.782459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.782398 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:50.783506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.783478 4793 scope.go:122] "RemoveContainer" containerID="24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669" Oct 30 04:20:50.783950 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:20:50.783927 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669\": container with ID starting with 24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669 not found: ID does not exist" containerID="24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669" Oct 30 04:20:50.784093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.784071 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669"} err="failed to get container status \"24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669\": rpc error: code = NotFound desc = could not find container \"24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669\": container with ID starting with 24ce31c1a467bf10eec4fb258a5cd49d6869615c54f8b7f9a763768cd3d46669 not found: ID does not exist" Oct 30 04:20:50.796912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.796886 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-tmp-dir\") pod \"5b193949-4e87-4992-8247-61849a043981\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " Oct 30 04:20:50.797012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.796954 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-home\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-home\") pod \"5b193949-4e87-4992-8247-61849a043981\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " Oct 30 04:20:50.797104 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.797090 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-dshm\") pod \"5b193949-4e87-4992-8247-61849a043981\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " Oct 30 04:20:50.797164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.797119 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-tmp-dir" pod "5b193949-4e87-4992-8247-61849a043981" (UID: "5b193949-4e87-4992-8247-61849a043981"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:20:50.797164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.797127 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b193949-4e87-4992-8247-61849a043981-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b193949-4e87-4992-8247-61849a043981-tls-certs\") pod \"5b193949-4e87-4992-8247-61849a043981\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " Oct 30 04:20:50.797273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.797219 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-model-cache\") pod \"5b193949-4e87-4992-8247-61849a043981\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " Oct 30 04:20:50.797273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.797243 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-home" pod "5b193949-4e87-4992-8247-61849a043981" (UID: "5b193949-4e87-4992-8247-61849a043981"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:20:50.797375 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.797345 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b193949-4e87-4992-8247-61849a043981-kube-api-access-sl4xf\" (UniqueName: \"kubernetes.io/projected/5b193949-4e87-4992-8247-61849a043981-kube-api-access-sl4xf\") pod \"5b193949-4e87-4992-8247-61849a043981\" (UID: \"5b193949-4e87-4992-8247-61849a043981\") " Oct 30 04:20:50.797557 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.797531 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-model-cache" pod "5b193949-4e87-4992-8247-61849a043981" (UID: "5b193949-4e87-4992-8247-61849a043981"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:20:50.797916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.797896 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:20:50.797916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.797917 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:20:50.798052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.797926 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:20:50.799393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.799369 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-dshm" pod "5b193949-4e87-4992-8247-61849a043981" (UID: "5b193949-4e87-4992-8247-61849a043981"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:20:50.799937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.799918 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b193949-4e87-4992-8247-61849a043981-kube-api-access-sl4xf" pod "5b193949-4e87-4992-8247-61849a043981" (UID: "5b193949-4e87-4992-8247-61849a043981"). InnerVolumeSpecName "kube-api-access-sl4xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:20:50.801855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.801830 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b193949-4e87-4992-8247-61849a043981-tls-certs" pod "5b193949-4e87-4992-8247-61849a043981" (UID: "5b193949-4e87-4992-8247-61849a043981"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:20:50.899341 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.899310 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b193949-4e87-4992-8247-61849a043981-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:20:50.899341 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.899335 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b193949-4e87-4992-8247-61849a043981-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:20:50.899341 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:50.899347 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl4xf\" (UniqueName: \"kubernetes.io/projected/5b193949-4e87-4992-8247-61849a043981-kube-api-access-sl4xf\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:20:51.087740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.087658 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8"] Oct 30 04:20:51.090736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.090709 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6796b1/e2e-conflict-a-54a3def2-kserve-696cdf765-5p4f8"] Oct 30 04:20:51.344753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.344672 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w"] Oct 30 04:20:51.345208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.345193 4793 cpu_manager.go:414] "containerMap: removing container" podUID="5b193949-4e87-4992-8247-61849a043981" containerName="main" Oct 30 04:20:51.345253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.345208 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b193949-4e87-4992-8247-61849a043981" containerName="main" Oct 30 04:20:51.345319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.345310 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b193949-4e87-4992-8247-61849a043981" containerName="main" Oct 30 04:20:51.350386 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.350362 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.380589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.373201 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w"] Oct 30 04:20:51.404805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.404766 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.404948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.404820 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.404990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.404949 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.405064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.405001 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.405180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.405146 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.405233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.405216 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.405265 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.405251 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.405384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.405368 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sv5c\" (UniqueName: \"kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-kube-api-access-8sv5c\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.405574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.405409 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.507227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.507198 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sv5c\" (UniqueName: \"kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-kube-api-access-8sv5c\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.507708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.507235 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.507708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.507294 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.507708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.507314 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.507708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.507341 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.507708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.507383 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.507708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.507475 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.507708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.507529 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.507708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.507557 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.508117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.507767 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.508117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.507854 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.508117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.508024 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.508117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.508081 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.508367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.508341 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.509953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.509930 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.510191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.510171 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.530684 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.530657 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.530924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.530905 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sv5c\" (UniqueName: \"kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-kube-api-access-8sv5c\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.667140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.667073 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:51.966298 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.966247 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:52.005714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:51.997524 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w"] Oct 30 04:20:52.013328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:52.013292 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:20:52.013483 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:52.013363 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:20:52.013483 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:52.013394 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:20:52.531767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:52.531726 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:52.772441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:52.769733 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:52.803147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:52.803049 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" event={"ID":"f4f3ab06-b4e7-4778-83f6-c43482adb47a","Type":"ContainerStarted","Data":"28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4"} Oct 30 04:20:52.803147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:52.803142 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" event={"ID":"f4f3ab06-b4e7-4778-83f6-c43482adb47a","Type":"ContainerStarted","Data":"693afdc0c4f6c28566ab6023792651535bdf93aa7afd84da91689f3ab1e8611d"} Oct 30 04:20:52.847460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:52.847433 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b193949-4e87-4992-8247-61849a043981" path="/var/lib/kubelet/pods/5b193949-4e87-4992-8247-61849a043981/volumes" Oct 30 04:20:52.892322 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:52.892265 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" podStartSLOduration=1.892250647 podStartE2EDuration="1.892250647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:20:52.891363122 +0000 UTC m=+1376.177237898" watchObservedRunningTime="2026-10-30 04:20:52.892250647 +0000 UTC m=+1376.178125417" Oct 30 04:20:53.670442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:53.667794 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:53.677445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:53.674532 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 04:20:53.677445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:53.674593 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" podUID="f4f3ab06-b4e7-4778-83f6-c43482adb47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 04:20:54.269664 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:54.269123 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:54.672501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:54.668737 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 04:20:54.672501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:54.668801 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" podUID="f4f3ab06-b4e7-4778-83f6-c43482adb47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 04:20:55.674443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:55.674355 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 04:20:55.674443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:55.674429 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" podUID="f4f3ab06-b4e7-4778-83f6-c43482adb47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 04:20:55.981752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:55.981722 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:56.667838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:56.667800 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 04:20:56.668118 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:56.668090 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" podUID="f4f3ab06-b4e7-4778-83f6-c43482adb47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 04:20:57.671579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:57.671537 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 04:20:57.672237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:57.672205 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" podUID="f4f3ab06-b4e7-4778-83f6-c43482adb47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 04:20:57.733592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:57.733551 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:58.667706 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:58.667668 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 04:20:58.667886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:58.667732 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" podUID="f4f3ab06-b4e7-4778-83f6-c43482adb47a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 04:20:59.401007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:59.400964 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:20:59.671853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:59.671781 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:59.672066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:59.672047 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:20:59.672762 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:20:59.672741 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 04:21:01.086248 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:01.086213 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:02.734544 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:02.734507 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:04.468245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:04.468206 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:06.160724 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:06.160612 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:07.755676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:07.755639 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:09.502856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:09.502757 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:11.220577 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:21:11.220538 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:21:11.316934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:11.316822 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:13.025024 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:13.024984 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:14.713267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:14.713229 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:16.327137 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:16.327099 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:18.010438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:18.010386 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:19.620574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:19.620535 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:19.660539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:19.660511 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57"] Oct 30 04:21:19.660741 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:19.660720 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" podUID="21b0ea95-77b3-46d5-9a4e-dca7b91efa76" containerName="istio-proxy" containerID="cri-o://ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8" gracePeriod=30 Oct 30 04:21:21.317143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:21.317100 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:23.014755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.014714 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:23.445269 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.445236 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn"] Oct 30 04:21:23.449878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.449854 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.483026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.482994 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn"] Oct 30 04:21:23.508531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.508497 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stqz4\" (UniqueName: \"kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-kube-api-access-stqz4\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.508684 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.508659 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.508826 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.508806 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.508862 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.508840 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f256eec9-b172-444a-b3fe-a311bf865e01-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.508908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.508866 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f256eec9-b172-444a-b3fe-a311bf865e01-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.509001 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.508978 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.509051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.509036 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.509182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.509153 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.509256 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.509213 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.610675 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.610632 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.610845 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.610700 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stqz4\" (UniqueName: \"kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-kube-api-access-stqz4\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.610845 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.610768 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.610962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.610869 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.610962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.610897 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f256eec9-b172-444a-b3fe-a311bf865e01-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.610962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.610925 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f256eec9-b172-444a-b3fe-a311bf865e01-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.611119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.610958 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.611119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.611008 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.611119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.611075 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.611119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.611094 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.611289 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.611130 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.611621 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.611594 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.611966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.611942 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.612740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.612708 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f256eec9-b172-444a-b3fe-a311bf865e01-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.621192 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.621164 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.622993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.622969 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f256eec9-b172-444a-b3fe-a311bf865e01-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.634824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.634805 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.634979 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.634962 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stqz4\" (UniqueName: \"kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-kube-api-access-stqz4\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:23.766437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:23.766361 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:24.083079 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:21:24.083046 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf256eec9_b172_444a_b3fe_a311bf865e01.slice/crio-174595ba57459ecbc8fcddd200932e1c4870b91cbc2d803e09a07e34ed4acaac WatchSource:0}: Error finding container 174595ba57459ecbc8fcddd200932e1c4870b91cbc2d803e09a07e34ed4acaac: Status 404 returned error can't find the container with id 174595ba57459ecbc8fcddd200932e1c4870b91cbc2d803e09a07e34ed4acaac Oct 30 04:21:24.083079 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:24.083070 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn"] Oct 30 04:21:24.085475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:24.085442 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:21:24.085587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:24.085516 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:21:24.085587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:24.085563 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:21:24.783203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:24.783136 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:24.982091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:24.981954 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:21:25.028710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.028680 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-kube-api-access-7rbdf\" (UniqueName: \"kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-kube-api-access-7rbdf\") pod \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " Oct 30 04:21:25.028885 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.028736 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-token\" (UniqueName: \"kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-token\") pod \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " Oct 30 04:21:25.028885 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.028766 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-data\") pod \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " Oct 30 04:21:25.028885 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.028868 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istiod-ca-cert\") pod \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " Oct 30 04:21:25.029075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.028970 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-podinfo\") pod \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " Oct 30 04:21:25.029134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029064 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-data" pod "21b0ea95-77b3-46d5-9a4e-dca7b91efa76" (UID: "21b0ea95-77b3-46d5-9a4e-dca7b91efa76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:25.029178 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029147 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-envoy\") pod \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " Oct 30 04:21:25.029214 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029182 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-certs\") pod \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " Oct 30 04:21:25.029258 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029219 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istiod-ca-cert" pod "21b0ea95-77b3-46d5-9a4e-dca7b91efa76" (UID: "21b0ea95-77b3-46d5-9a4e-dca7b91efa76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:21:25.029258 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029232 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-credential-socket\") pod \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " Oct 30 04:21:25.029363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029268 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-socket\") pod \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\" (UID: \"21b0ea95-77b3-46d5-9a4e-dca7b91efa76\") " Oct 30 04:21:25.029915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029492 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-certs" pod "21b0ea95-77b3-46d5-9a4e-dca7b91efa76" (UID: "21b0ea95-77b3-46d5-9a4e-dca7b91efa76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:25.029915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029525 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-credential-socket" pod "21b0ea95-77b3-46d5-9a4e-dca7b91efa76" (UID: "21b0ea95-77b3-46d5-9a4e-dca7b91efa76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:25.029915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029549 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-socket" pod "21b0ea95-77b3-46d5-9a4e-dca7b91efa76" (UID: "21b0ea95-77b3-46d5-9a4e-dca7b91efa76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:25.029915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029815 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:21:25.029915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029838 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:21:25.029915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029853 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:21:25.029915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029868 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:21:25.029915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.029882 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:21:25.031128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.031102 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-token" pod "21b0ea95-77b3-46d5-9a4e-dca7b91efa76" (UID: "21b0ea95-77b3-46d5-9a4e-dca7b91efa76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:25.031239 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.031190 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-kube-api-access-7rbdf" pod "21b0ea95-77b3-46d5-9a4e-dca7b91efa76" (UID: "21b0ea95-77b3-46d5-9a4e-dca7b91efa76"). InnerVolumeSpecName "kube-api-access-7rbdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:21:25.031352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.031334 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-envoy" pod "21b0ea95-77b3-46d5-9a4e-dca7b91efa76" (UID: "21b0ea95-77b3-46d5-9a4e-dca7b91efa76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:21:25.032282 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.032262 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-podinfo" pod "21b0ea95-77b3-46d5-9a4e-dca7b91efa76" (UID: "21b0ea95-77b3-46d5-9a4e-dca7b91efa76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:21:25.093490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.093458 4793 generic.go:356] "Generic (PLEG): container finished" podID="21b0ea95-77b3-46d5-9a4e-dca7b91efa76" containerID="ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8" exitCode=0 Oct 30 04:21:25.093965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.093558 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" Oct 30 04:21:25.093965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.093567 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" event={"ID":"21b0ea95-77b3-46d5-9a4e-dca7b91efa76","Type":"ContainerDied","Data":"ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8"} Oct 30 04:21:25.093965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.093619 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57" event={"ID":"21b0ea95-77b3-46d5-9a4e-dca7b91efa76","Type":"ContainerDied","Data":"e3e6f20d9b023efd4f76a16649991700571de633474c8057901df0cfe7e32aa5"} Oct 30 04:21:25.093965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.093651 4793 scope.go:122] "RemoveContainer" containerID="ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8" Oct 30 04:21:25.095918 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.095886 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" event={"ID":"f256eec9-b172-444a-b3fe-a311bf865e01","Type":"ContainerStarted","Data":"f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8"} Oct 30 04:21:25.096043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.095949 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" event={"ID":"f256eec9-b172-444a-b3fe-a311bf865e01","Type":"ContainerStarted","Data":"174595ba57459ecbc8fcddd200932e1c4870b91cbc2d803e09a07e34ed4acaac"} Oct 30 04:21:25.110625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.110121 4793 scope.go:122] "RemoveContainer" containerID="ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8" Oct 30 04:21:25.110625 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:21:25.110450 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8\": container with ID starting with ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8 not found: ID does not exist" containerID="ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8" Oct 30 04:21:25.110625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.110487 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8"} err="failed to get container status \"ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8\": rpc error: code = NotFound desc = could not find container \"ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8\": container with ID starting with ccf36d09c62d77cf1eb57b3953b0651c40f5c9051f596c1b982fecc18df6b5e8 not found: ID does not exist" Oct 30 04:21:25.130676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.130654 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:21:25.130676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.130674 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rbdf\" (UniqueName: \"kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-kube-api-access-7rbdf\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:21:25.130844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.130682 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:21:25.130844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.130690 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21b0ea95-77b3-46d5-9a4e-dca7b91efa76-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:21:25.221449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.221382 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" podStartSLOduration=2.221364998 podStartE2EDuration="2.221364998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:25.183764568 +0000 UTC m=+1408.469639346" watchObservedRunningTime="2026-10-30 04:21:25.221364998 +0000 UTC m=+1408.507239773" Oct 30 04:21:25.223632 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.223610 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57"] Oct 30 04:21:25.235095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.235030 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-scw57"] Oct 30 04:21:25.778520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.771153 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:25.778982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.778949 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 04:21:25.779091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:25.779017 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" podUID="f256eec9-b172-444a-b3fe-a311bf865e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 04:21:26.437929 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:26.437853 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:26.773498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:26.769808 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 04:21:26.773498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:26.769884 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" podUID="f256eec9-b172-444a-b3fe-a311bf865e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 04:21:26.846255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:26.846225 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b0ea95-77b3-46d5-9a4e-dca7b91efa76" path="/var/lib/kubelet/pods/21b0ea95-77b3-46d5-9a4e-dca7b91efa76/volumes" Oct 30 04:21:27.778587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:27.778541 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 04:21:27.778962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:27.778614 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" podUID="f256eec9-b172-444a-b3fe-a311bf865e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 04:21:28.238338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:28.238303 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:28.773578 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:28.773541 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 04:21:28.773998 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:28.773965 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" podUID="f256eec9-b172-444a-b3fe-a311bf865e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 04:21:29.772444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:29.768321 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 04:21:29.772444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:29.768383 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" podUID="f256eec9-b172-444a-b3fe-a311bf865e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 04:21:29.940566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:29.940519 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:30.966769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:30.966732 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:30.967628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:30.967556 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:30.968935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:30.968912 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 04:21:31.610095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:31.610060 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:33.219383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:33.219348 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:34.937932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:34.937894 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:36.595929 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:36.595897 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:37.022381 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:37.022347 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:37.239621 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:37.239585 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:38.288010 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:38.287967 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:39.984225 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:39.984186 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:41.219547 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:21:41.219516 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:21:41.601152 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:41.601059 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:42.865906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.865874 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs"] Oct 30 04:21:42.866718 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.866701 4793 cpu_manager.go:414] "containerMap: removing container" podUID="21b0ea95-77b3-46d5-9a4e-dca7b91efa76" containerName="istio-proxy" Oct 30 04:21:42.866774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.866722 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b0ea95-77b3-46d5-9a4e-dca7b91efa76" containerName="istio-proxy" Oct 30 04:21:42.866894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.866882 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b0ea95-77b3-46d5-9a4e-dca7b91efa76" containerName="istio-proxy" Oct 30 04:21:42.872485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.872462 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:42.887386 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.887361 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs"] Oct 30 04:21:42.894791 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.894765 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-36bbac\"/\"openshift-service-ca.crt\"" Oct 30 04:21:42.899889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.899867 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-36bbac\"/\"default-dockercfg-llsp2\"" Oct 30 04:21:42.899889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.899880 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-36bbac\"/\"kube-root-ca.crt\"" Oct 30 04:21:42.900086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.900070 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-36bbac\"/\"distinct-w0-36bbac-kserve-self-signed-certs\"" Oct 30 04:21:42.942066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.942038 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-dshm\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:42.942162 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.942105 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndxg6\" (UniqueName: \"kubernetes.io/projected/439ce07a-ea3d-4b97-a8ed-7655a840c873-kube-api-access-ndxg6\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:42.942239 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.942219 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439ce07a-ea3d-4b97-a8ed-7655a840c873-tls-certs\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:42.942299 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.942258 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-model-cache\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:42.942359 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.942329 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-home\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:42.942564 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:42.942540 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-tmp-dir\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.020606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.020583 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:43.044267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.044238 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-tmp-dir\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.044501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.044357 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-dshm\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.044501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.044431 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndxg6\" (UniqueName: \"kubernetes.io/projected/439ce07a-ea3d-4b97-a8ed-7655a840c873-kube-api-access-ndxg6\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.044501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.044491 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439ce07a-ea3d-4b97-a8ed-7655a840c873-tls-certs\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.044685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.044533 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-model-cache\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.044685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.044558 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-home\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.044938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.044913 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-tmp-dir\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.045233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.045195 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-home\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.053798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.053455 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-model-cache\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.053798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.053738 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-dshm\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.068412 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.068365 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439ce07a-ea3d-4b97-a8ed-7655a840c873-tls-certs\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.104363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.104229 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndxg6\" (UniqueName: \"kubernetes.io/projected/439ce07a-ea3d-4b97-a8ed-7655a840c873-kube-api-access-ndxg6\") pod \"distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.187813 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.187777 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:43.274467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.274340 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:43.513657 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:21:43.513623 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod439ce07a_ea3d_4b97_a8ed_7655a840c873.slice/crio-9e0bdf80162895a9b6eccf80c4e2d81b06540228b459a6a7a81ad4670e4406c1 WatchSource:0}: Error finding container 9e0bdf80162895a9b6eccf80c4e2d81b06540228b459a6a7a81ad4670e4406c1: Status 404 returned error can't find the container with id 9e0bdf80162895a9b6eccf80c4e2d81b06540228b459a6a7a81ad4670e4406c1 Oct 30 04:21:43.514505 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:43.514485 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs"] Oct 30 04:21:44.241877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:44.241841 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" event={"ID":"439ce07a-ea3d-4b97-a8ed-7655a840c873","Type":"ContainerStarted","Data":"0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12"} Oct 30 04:21:44.241877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:44.241884 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" event={"ID":"439ce07a-ea3d-4b97-a8ed-7655a840c873","Type":"ContainerStarted","Data":"9e0bdf80162895a9b6eccf80c4e2d81b06540228b459a6a7a81ad4670e4406c1"} Oct 30 04:21:44.287638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:44.287585 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" podStartSLOduration=2.287571985 podStartE2EDuration="2.287571985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:21:44.283587573 +0000 UTC m=+1427.569462382" watchObservedRunningTime="2026-10-30 04:21:44.287571985 +0000 UTC m=+1427.573446764" Oct 30 04:21:45.016831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:45.016795 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:45.188185 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:45.188143 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:46.703647 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:46.703609 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:48.320399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:48.320362 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:50.086723 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:50.086687 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:51.827768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:51.827732 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:52.697061 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:52.697022 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:53.188897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:53.188861 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:53.205565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:53.205540 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:53.332752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:53.332726 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 04:21:53.428260 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:53.428222 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:55.030488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:55.030448 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:56.695456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:56.695406 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:21:58.374087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:21:58.374041 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:00.112402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:00.112361 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:01.804204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:01.804104 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:03.430579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:03.430537 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:05.132945 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:05.132909 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:06.724078 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:06.724036 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:08.532760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:08.532719 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:10.208803 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:10.208766 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:11.218605 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:22:11.218572 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:22:11.920323 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:11.920283 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:13.033531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.033497 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4"] Oct 30 04:22:13.038733 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.038712 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.073229 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.073199 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4"] Oct 30 04:22:13.105221 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.105194 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.105342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.105224 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.105342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.105250 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt8rn\" (UniqueName: \"kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-kube-api-access-kt8rn\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.105444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.105346 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389a8074-a44c-42c7-8696-775753993e07-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.105444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.105375 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.105444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.105401 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389a8074-a44c-42c7-8696-775753993e07-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.105541 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.105490 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.105576 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.105555 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.105657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.105633 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.206555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.206527 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389a8074-a44c-42c7-8696-775753993e07-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.206696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.206562 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.206696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.206586 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389a8074-a44c-42c7-8696-775753993e07-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.206696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.206620 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.206867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.206704 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.206867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.206741 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.206975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.206905 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.206975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.206936 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.207083 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.206990 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt8rn\" (UniqueName: \"kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-kube-api-access-kt8rn\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.207083 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.207015 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.207308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.207283 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.207362 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.207341 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.207460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.207441 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.207772 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.207750 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389a8074-a44c-42c7-8696-775753993e07-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.209394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.209374 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.209729 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.209708 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389a8074-a44c-42c7-8696-775753993e07-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.233819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.233793 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.236544 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.236520 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt8rn\" (UniqueName: \"kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-kube-api-access-kt8rn\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.351926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.351853 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:13.617275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.617177 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:13.668298 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:22:13.668264 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389a8074_a44c_42c7_8696_775753993e07.slice/crio-84814657d80e5384c91ab13e54d63545e23415392cdf477905cb86ed7e2eec0e WatchSource:0}: Error finding container 84814657d80e5384c91ab13e54d63545e23415392cdf477905cb86ed7e2eec0e: Status 404 returned error can't find the container with id 84814657d80e5384c91ab13e54d63545e23415392cdf477905cb86ed7e2eec0e Oct 30 04:22:13.668992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.668944 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4"] Oct 30 04:22:13.670744 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.670707 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:22:13.670841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.670785 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:22:13.670841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:13.670834 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:22:14.489191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:14.489153 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" event={"ID":"389a8074-a44c-42c7-8696-775753993e07","Type":"ContainerStarted","Data":"d0c7e8c06b501788b64d1125aa680c2bb8493dfcf2652bb58ec734945fcf4792"} Oct 30 04:22:14.489711 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:14.489214 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" event={"ID":"389a8074-a44c-42c7-8696-775753993e07","Type":"ContainerStarted","Data":"84814657d80e5384c91ab13e54d63545e23415392cdf477905cb86ed7e2eec0e"} Oct 30 04:22:14.550777 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:14.550718 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" podStartSLOduration=1.550704177 podStartE2EDuration="1.550704177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:22:14.548229856 +0000 UTC m=+1457.834104634" watchObservedRunningTime="2026-10-30 04:22:14.550704177 +0000 UTC m=+1457.836578951" Oct 30 04:22:15.352204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:15.352172 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:15.356005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:15.355586 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 04:22:15.356005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:15.355679 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" podUID="389a8074-a44c-42c7-8696-775753993e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 04:22:15.421668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:15.421633 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:16.352762 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:16.352730 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 04:22:16.353160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:16.352790 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" podUID="389a8074-a44c-42c7-8696-775753993e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 04:22:17.052757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:17.052718 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:17.352732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:17.352644 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 04:22:17.352732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:17.352697 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" podUID="389a8074-a44c-42c7-8696-775753993e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 04:22:18.352250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:18.352216 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 04:22:18.352400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:18.352276 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" podUID="389a8074-a44c-42c7-8696-775753993e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 04:22:18.719710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:18.719670 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:19.352444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:19.352392 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 04:22:19.352616 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:19.352468 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" podUID="389a8074-a44c-42c7-8696-775753993e07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 04:22:20.543859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:20.543815 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:20.557681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:20.557657 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:20.558129 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:20.558103 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:20.560156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:20.560135 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:22:22.128665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:22.128629 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:23.715087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:23.715048 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:25.418851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:25.418814 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:27.022429 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:27.022378 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:28.717094 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:28.717056 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:30.426635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:30.426596 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:32.121915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:32.121880 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:33.825554 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:33.825517 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:35.538529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:35.538484 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:37.196296 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:37.196258 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:38.922654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:38.922613 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:40.514899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:40.514796 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:41.219643 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:22:41.219607 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:22:42.217302 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:42.217266 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:43.918262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:43.918221 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:45.636127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:45.636088 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:47.280483 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:47.280453 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:48.901895 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:48.901855 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:49.898892 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:49.898861 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g"] Oct 30 04:22:49.899101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:49.899057 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" podUID="2e816986-f54e-44d3-b431-0d09094fa6ff" containerName="istio-proxy" containerID="cri-o://e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54" gracePeriod=30 Oct 30 04:22:50.624087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:50.624049 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:52.287814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:52.287773 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:52.882647 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:52.882615 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:54.078505 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:54.078474 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:55.174374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.174320 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:22:55.220805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.220777 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-podinfo\") pod \"2e816986-f54e-44d3-b431-0d09094fa6ff\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " Oct 30 04:22:55.220963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.220838 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-certs\") pod \"2e816986-f54e-44d3-b431-0d09094fa6ff\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " Oct 30 04:22:55.220963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.220867 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-envoy\") pod \"2e816986-f54e-44d3-b431-0d09094fa6ff\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " Oct 30 04:22:55.220963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.220910 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e816986-f54e-44d3-b431-0d09094fa6ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e816986-f54e-44d3-b431-0d09094fa6ff-istiod-ca-cert\") pod \"2e816986-f54e-44d3-b431-0d09094fa6ff\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " Oct 30 04:22:55.221133 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.220966 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-data\") pod \"2e816986-f54e-44d3-b431-0d09094fa6ff\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " Oct 30 04:22:55.221133 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.221006 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-token\" (UniqueName: \"kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-token\") pod \"2e816986-f54e-44d3-b431-0d09094fa6ff\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " Oct 30 04:22:55.221133 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.221043 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-socket\") pod \"2e816986-f54e-44d3-b431-0d09094fa6ff\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " Oct 30 04:22:55.221133 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.221112 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-certs" pod "2e816986-f54e-44d3-b431-0d09094fa6ff" (UID: "2e816986-f54e-44d3-b431-0d09094fa6ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:22:55.221339 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.221127 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-credential-socket\") pod \"2e816986-f54e-44d3-b431-0d09094fa6ff\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " Oct 30 04:22:55.221339 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.221235 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-kube-api-access-24nns\" (UniqueName: \"kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-kube-api-access-24nns\") pod \"2e816986-f54e-44d3-b431-0d09094fa6ff\" (UID: \"2e816986-f54e-44d3-b431-0d09094fa6ff\") " Oct 30 04:22:55.221474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.221437 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-socket" pod "2e816986-f54e-44d3-b431-0d09094fa6ff" (UID: "2e816986-f54e-44d3-b431-0d09094fa6ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:22:55.221848 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.221809 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-credential-socket" pod "2e816986-f54e-44d3-b431-0d09094fa6ff" (UID: "2e816986-f54e-44d3-b431-0d09094fa6ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:22:55.221975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.221848 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-data" pod "2e816986-f54e-44d3-b431-0d09094fa6ff" (UID: "2e816986-f54e-44d3-b431-0d09094fa6ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:22:55.221975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.221935 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:22:55.221975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.221959 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:22:55.221975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.221973 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:22:55.222205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.221994 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e816986-f54e-44d3-b431-0d09094fa6ff-istiod-ca-cert" pod "2e816986-f54e-44d3-b431-0d09094fa6ff" (UID: "2e816986-f54e-44d3-b431-0d09094fa6ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:22:55.224339 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.224311 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-kube-api-access-24nns" pod "2e816986-f54e-44d3-b431-0d09094fa6ff" (UID: "2e816986-f54e-44d3-b431-0d09094fa6ff"). InnerVolumeSpecName "kube-api-access-24nns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:22:55.224339 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.224325 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-token" pod "2e816986-f54e-44d3-b431-0d09094fa6ff" (UID: "2e816986-f54e-44d3-b431-0d09094fa6ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:22:55.224548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.224342 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-podinfo" pod "2e816986-f54e-44d3-b431-0d09094fa6ff" (UID: "2e816986-f54e-44d3-b431-0d09094fa6ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:22:55.224685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.224655 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-envoy" pod "2e816986-f54e-44d3-b431-0d09094fa6ff" (UID: "2e816986-f54e-44d3-b431-0d09094fa6ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:22:55.323011 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.322981 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:22:55.323011 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.323011 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:22:55.323210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.323020 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24nns\" (UniqueName: \"kubernetes.io/projected/2e816986-f54e-44d3-b431-0d09094fa6ff-kube-api-access-24nns\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:22:55.323210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.323029 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:22:55.323210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.323037 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e816986-f54e-44d3-b431-0d09094fa6ff-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:22:55.323210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.323045 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e816986-f54e-44d3-b431-0d09094fa6ff-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:22:55.778183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.775561 4793 generic.go:356] "Generic (PLEG): container finished" podID="2e816986-f54e-44d3-b431-0d09094fa6ff" containerID="e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54" exitCode=0 Oct 30 04:22:55.778183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.775664 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" event={"ID":"2e816986-f54e-44d3-b431-0d09094fa6ff","Type":"ContainerDied","Data":"e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54"} Oct 30 04:22:55.778183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.775714 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" event={"ID":"2e816986-f54e-44d3-b431-0d09094fa6ff","Type":"ContainerDied","Data":"5c9287bb4bef2b3667930133dda387ad57b9e69b45b38b6fc7f72a8bb01d0efe"} Oct 30 04:22:55.778183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.775743 4793 scope.go:122] "RemoveContainer" containerID="e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54" Oct 30 04:22:55.778183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.775914 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g" Oct 30 04:22:55.793049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.793024 4793 scope.go:122] "RemoveContainer" containerID="e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54" Oct 30 04:22:55.793437 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:22:55.793396 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54\": container with ID starting with e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54 not found: ID does not exist" containerID="e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54" Oct 30 04:22:55.793517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.793438 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54"} err="failed to get container status \"e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54\": rpc error: code = NotFound desc = could not find container \"e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54\": container with ID starting with e6d627d7df5b594cbd3dc22a5bbb0029200bc09bed80f8667d7c595a5ecf4e54 not found: ID does not exist" Oct 30 04:22:55.810916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.810885 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:55.832042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.832020 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g"] Oct 30 04:22:55.839869 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:55.839847 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-w6h4g"] Oct 30 04:22:56.854586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:56.854554 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e816986-f54e-44d3-b431-0d09094fa6ff" path="/var/lib/kubelet/pods/2e816986-f54e-44d3-b431-0d09094fa6ff/volumes" Oct 30 04:22:57.502644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:57.502607 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:22:59.135187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:22:59.135151 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:00.799869 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:00.799678 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:02.520686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:02.520640 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:04.221046 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:04.221006 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:06.004191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:06.004156 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:07.725767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:07.725726 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:09.225959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:09.225928 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:10.895307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:10.895274 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:11.219489 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:23:11.219448 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:23:12.618119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:12.618076 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:14.408081 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:14.408045 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:16.145650 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:16.145606 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:17.772333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:17.772290 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:18.607964 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:18.607926 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:19.035287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.035259 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t"] Oct 30 04:23:19.035853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.035838 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2e816986-f54e-44d3-b431-0d09094fa6ff" containerName="istio-proxy" Oct 30 04:23:19.035897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.035854 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e816986-f54e-44d3-b431-0d09094fa6ff" containerName="istio-proxy" Oct 30 04:23:19.035951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.035942 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e816986-f54e-44d3-b431-0d09094fa6ff" containerName="istio-proxy" Oct 30 04:23:19.039022 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.039006 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.062047 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.062027 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1a0d3427-openshift-default-dockercfg-hbjqw\"" Oct 30 04:23:19.079028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.079000 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t"] Oct 30 04:23:19.181713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.181679 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:19.191234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.191205 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a10528f-fdbe-4c27-8b64-67adff5df547-istiod-ca-cert\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.191569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.191551 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-token\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.191771 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.191754 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-podinfo\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.192047 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.192030 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-credential-socket\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.192292 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.192275 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-envoy\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.192471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.192455 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-certs\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.192683 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.192666 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-socket\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.192894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.192875 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvrjq\" (UniqueName: \"kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-kube-api-access-dvrjq\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.193287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.193270 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-data\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.295255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295160 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-credential-socket\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.295255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295228 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-envoy\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.295255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295249 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-certs\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.295579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295277 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-socket\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.295579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295310 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvrjq\" (UniqueName: \"kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-kube-api-access-dvrjq\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.295579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295464 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-data\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.295579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295515 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a10528f-fdbe-4c27-8b64-67adff5df547-istiod-ca-cert\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.295579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295554 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-token\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.295847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295586 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-podinfo\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.295847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295678 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-credential-socket\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.295847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295690 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-certs\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.295847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295757 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-socket\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.296057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.295931 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-data\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.296378 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.296356 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a10528f-fdbe-4c27-8b64-67adff5df547-istiod-ca-cert\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.297839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.297808 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-envoy\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.298128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.298108 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-podinfo\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.320208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.320163 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-token\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.320359 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.320305 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvrjq\" (UniqueName: \"kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-kube-api-access-dvrjq\") pod \"e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.352705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.352604 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:19.528591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.528548 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:19.671517 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:23:19.671467 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a10528f_fdbe_4c27_8b64_67adff5df547.slice/crio-2014ac4b10893e7b1f85a643a0291112caf9b9f468c223eb9acae288b91ded34 WatchSource:0}: Error finding container 2014ac4b10893e7b1f85a643a0291112caf9b9f468c223eb9acae288b91ded34: Status 404 returned error can't find the container with id 2014ac4b10893e7b1f85a643a0291112caf9b9f468c223eb9acae288b91ded34 Oct 30 04:23:19.671845 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.671823 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t"] Oct 30 04:23:19.673828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.673795 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:23:19.673906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.673863 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:23:19.673906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.673895 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:23:19.763713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.763680 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:19.936477 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.936411 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" event={"ID":"0a10528f-fdbe-4c27-8b64-67adff5df547","Type":"ContainerStarted","Data":"c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7"} Oct 30 04:23:19.936666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:19.936510 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" event={"ID":"0a10528f-fdbe-4c27-8b64-67adff5df547","Type":"ContainerStarted","Data":"2014ac4b10893e7b1f85a643a0291112caf9b9f468c223eb9acae288b91ded34"} Oct 30 04:23:20.036297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:20.036204 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" podStartSLOduration=1.036187026 podStartE2EDuration="1.036187026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:23:20.013662275 +0000 UTC m=+1523.299537051" watchObservedRunningTime="2026-10-30 04:23:20.036187026 +0000 UTC m=+1523.322061817" Oct 30 04:23:20.042043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:20.042010 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8"] Oct 30 04:23:20.042796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:20.042760 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" containerID="cri-o://380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00" gracePeriod=30 Oct 30 04:23:20.353082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:20.353013 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:20.358689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:20.358663 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:20.943577 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:20.943542 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:20.944470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:20.944449 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:21.315327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:21.315226 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:22.588068 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:22.588028 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:22.726954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:22.726923 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-9vqv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:23:22.727136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:22.726985 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:23:22.806347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:22.806311 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:23.142886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:23.142850 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:24.212590 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.212558 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:24.221849 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.221824 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp"] Oct 30 04:23:24.226332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.226312 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" Oct 30 04:23:24.246052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.246026 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp"] Oct 30 04:23:24.253495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.253476 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1a0d3427\"" Oct 30 04:23:24.364475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.363288 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9857c7f-c96b-4057-8c34-7359cf82ffef-plugins-config-volume\") pod \"payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp\" (UID: \"d9857c7f-c96b-4057-8c34-7359cf82ffef\") " pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" Oct 30 04:23:24.364475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.363573 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzg98\" (UniqueName: \"kubernetes.io/projected/d9857c7f-c96b-4057-8c34-7359cf82ffef-kube-api-access-mzg98\") pod \"payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp\" (UID: \"d9857c7f-c96b-4057-8c34-7359cf82ffef\") " pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" Oct 30 04:23:24.451485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.451188 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:24.474522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.465922 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzg98\" (UniqueName: \"kubernetes.io/projected/d9857c7f-c96b-4057-8c34-7359cf82ffef-kube-api-access-mzg98\") pod \"payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp\" (UID: \"d9857c7f-c96b-4057-8c34-7359cf82ffef\") " pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" Oct 30 04:23:24.474522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.466010 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9857c7f-c96b-4057-8c34-7359cf82ffef-plugins-config-volume\") pod \"payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp\" (UID: \"d9857c7f-c96b-4057-8c34-7359cf82ffef\") " pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" Oct 30 04:23:24.474522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.466959 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9857c7f-c96b-4057-8c34-7359cf82ffef-plugins-config-volume\") pod \"payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp\" (UID: \"d9857c7f-c96b-4057-8c34-7359cf82ffef\") " pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" Oct 30 04:23:24.499954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.499916 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzg98\" (UniqueName: \"kubernetes.io/projected/d9857c7f-c96b-4057-8c34-7359cf82ffef-kube-api-access-mzg98\") pod \"payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp\" (UID: \"d9857c7f-c96b-4057-8c34-7359cf82ffef\") " pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" Oct 30 04:23:24.540990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.540957 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" Oct 30 04:23:24.814644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.814549 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:24.851920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.851893 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp"] Oct 30 04:23:24.852103 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:23:24.852080 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9857c7f_c96b_4057_8c34_7359cf82ffef.slice/crio-1e3c4c7d20e0556ba631b6c94397701fb61472c635a057ed33eb46fca568a26f WatchSource:0}: Error finding container 1e3c4c7d20e0556ba631b6c94397701fb61472c635a057ed33eb46fca568a26f: Status 404 returned error can't find the container with id 1e3c4c7d20e0556ba631b6c94397701fb61472c635a057ed33eb46fca568a26f Oct 30 04:23:24.974484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.974452 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" event={"ID":"d9857c7f-c96b-4057-8c34-7359cf82ffef","Type":"ContainerStarted","Data":"03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea"} Oct 30 04:23:24.974622 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.974506 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" event={"ID":"d9857c7f-c96b-4057-8c34-7359cf82ffef","Type":"ContainerStarted","Data":"1e3c4c7d20e0556ba631b6c94397701fb61472c635a057ed33eb46fca568a26f"} Oct 30 04:23:24.974693 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:24.974630 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" Oct 30 04:23:25.018989 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.018941 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" podStartSLOduration=1.018927452 podStartE2EDuration="1.018927452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:23:25.015995372 +0000 UTC m=+1528.301870149" watchObservedRunningTime="2026-10-30 04:23:25.018927452 +0000 UTC m=+1528.304802227" Oct 30 04:23:25.328246 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.328165 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:23:25.477121 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477091 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-data\") pod \"734b5f28-3d4f-4674-b509-0fb356ee779b\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " Oct 30 04:23:25.477294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477130 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-socket\") pod \"734b5f28-3d4f-4674-b509-0fb356ee779b\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " Oct 30 04:23:25.477294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477200 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-podinfo\") pod \"734b5f28-3d4f-4674-b509-0fb356ee779b\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " Oct 30 04:23:25.477294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477249 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734b5f28-3d4f-4674-b509-0fb356ee779b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734b5f28-3d4f-4674-b509-0fb356ee779b-istiod-ca-cert\") pod \"734b5f28-3d4f-4674-b509-0fb356ee779b\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " Oct 30 04:23:25.477481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477311 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-token\" (UniqueName: \"kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-token\") pod \"734b5f28-3d4f-4674-b509-0fb356ee779b\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " Oct 30 04:23:25.477481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477355 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-kube-api-access-55r6s\" (UniqueName: \"kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-kube-api-access-55r6s\") pod \"734b5f28-3d4f-4674-b509-0fb356ee779b\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " Oct 30 04:23:25.477481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477404 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-data" pod "734b5f28-3d4f-4674-b509-0fb356ee779b" (UID: "734b5f28-3d4f-4674-b509-0fb356ee779b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:23:25.477481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477434 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-certs\") pod \"734b5f28-3d4f-4674-b509-0fb356ee779b\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " Oct 30 04:23:25.477689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477550 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-envoy\") pod \"734b5f28-3d4f-4674-b509-0fb356ee779b\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " Oct 30 04:23:25.477689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477595 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-socket" pod "734b5f28-3d4f-4674-b509-0fb356ee779b" (UID: "734b5f28-3d4f-4674-b509-0fb356ee779b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:23:25.477689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477600 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-credential-socket\") pod \"734b5f28-3d4f-4674-b509-0fb356ee779b\" (UID: \"734b5f28-3d4f-4674-b509-0fb356ee779b\") " Oct 30 04:23:25.478151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477908 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-certs" pod "734b5f28-3d4f-4674-b509-0fb356ee779b" (UID: "734b5f28-3d4f-4674-b509-0fb356ee779b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:23:25.478151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.477928 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734b5f28-3d4f-4674-b509-0fb356ee779b-istiod-ca-cert" pod "734b5f28-3d4f-4674-b509-0fb356ee779b" (UID: "734b5f28-3d4f-4674-b509-0fb356ee779b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:23:25.478151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.478109 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-credential-socket" pod "734b5f28-3d4f-4674-b509-0fb356ee779b" (UID: "734b5f28-3d4f-4674-b509-0fb356ee779b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:23:25.478414 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.478286 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/734b5f28-3d4f-4674-b509-0fb356ee779b-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:23:25.478414 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.478309 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:23:25.478414 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.478323 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:23:25.478414 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.478336 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:23:25.478414 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.478349 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:23:25.480095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.480059 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-kube-api-access-55r6s" pod "734b5f28-3d4f-4674-b509-0fb356ee779b" (UID: "734b5f28-3d4f-4674-b509-0fb356ee779b"). InnerVolumeSpecName "kube-api-access-55r6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:23:25.480191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.480151 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-token" pod "734b5f28-3d4f-4674-b509-0fb356ee779b" (UID: "734b5f28-3d4f-4674-b509-0fb356ee779b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:23:25.480191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.480161 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-podinfo" pod "734b5f28-3d4f-4674-b509-0fb356ee779b" (UID: "734b5f28-3d4f-4674-b509-0fb356ee779b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:23:25.480353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.480337 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-envoy" pod "734b5f28-3d4f-4674-b509-0fb356ee779b" (UID: "734b5f28-3d4f-4674-b509-0fb356ee779b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:23:25.579344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.579269 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:23:25.579344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.579303 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:23:25.579344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.579318 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:23:25.579344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.579334 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55r6s\" (UniqueName: \"kubernetes.io/projected/734b5f28-3d4f-4674-b509-0fb356ee779b-kube-api-access-55r6s\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:23:25.982637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.982604 4793 generic.go:356] "Generic (PLEG): container finished" podID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerID="380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00" exitCode=0 Oct 30 04:23:25.982829 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.982721 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" Oct 30 04:23:25.982829 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.982742 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" event={"ID":"734b5f28-3d4f-4674-b509-0fb356ee779b","Type":"ContainerDied","Data":"380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00"} Oct 30 04:23:25.982829 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.982798 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8" event={"ID":"734b5f28-3d4f-4674-b509-0fb356ee779b","Type":"ContainerDied","Data":"5664b309cf46bbd004efd90323238f6aba7582c091c373bfde9105db2b915361"} Oct 30 04:23:25.982829 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.982820 4793 scope.go:122] "RemoveContainer" containerID="380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00" Oct 30 04:23:25.995847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:25.995830 4793 scope.go:122] "RemoveContainer" containerID="80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8" Oct 30 04:23:26.005078 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:26.005060 4793 scope.go:122] "RemoveContainer" containerID="380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00" Oct 30 04:23:26.005322 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:23:26.005300 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00\": container with ID starting with 380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00 not found: ID does not exist" containerID="380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00" Oct 30 04:23:26.005385 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:26.005330 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00"} err="failed to get container status \"380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00\": rpc error: code = NotFound desc = could not find container \"380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00\": container with ID starting with 380f428fe74f3491a2ef55a21590ff9cb834728e7a3ab89991840007f6464d00 not found: ID does not exist" Oct 30 04:23:26.005385 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:26.005359 4793 scope.go:122] "RemoveContainer" containerID="80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8" Oct 30 04:23:26.005606 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:23:26.005589 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8\": container with ID starting with 80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8 not found: ID does not exist" containerID="80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8" Oct 30 04:23:26.005649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:26.005612 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8"} err="failed to get container status \"80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8\": rpc error: code = NotFound desc = could not find container \"80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8\": container with ID starting with 80f44a55b061bf7f1493c3772294e94743ba0a3ffe2009310df009a5f7fffdd8 not found: ID does not exist" Oct 30 04:23:26.037461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:26.037438 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8"] Oct 30 04:23:26.051543 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:26.051519 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-9vqv8"] Oct 30 04:23:26.513202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:26.513165 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:26.856548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:26.856461 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" path="/var/lib/kubelet/pods/734b5f28-3d4f-4674-b509-0fb356ee779b/volumes" Oct 30 04:23:27.714347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:27.714311 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:27.970042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:27.969961 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:28.329245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:28.329153 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:29.099648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:29.099610 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:29.450831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:29.450799 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:30.216016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.215983 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:30.660441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.660396 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p"] Oct 30 04:23:30.660968 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.660952 4793 cpu_manager.go:414] "containerMap: removing container" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" Oct 30 04:23:30.661010 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.660970 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" Oct 30 04:23:30.661010 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.660981 4793 cpu_manager.go:414] "containerMap: removing container" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" Oct 30 04:23:30.661010 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.660986 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" Oct 30 04:23:30.661100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.661070 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" Oct 30 04:23:30.661100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.661079 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="734b5f28-3d4f-4674-b509-0fb356ee779b" containerName="istio-proxy" Oct 30 04:23:30.666143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.666123 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:30.682241 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.682216 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p"] Oct 30 04:23:30.690411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.690371 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:23:30.690569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.690446 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:23:30.690569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.690459 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:23:30.690569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.690535 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:23:30.845969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.845940 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mkc\" (UniqueName: \"kubernetes.io/projected/3149dd11-8250-4113-8498-88d836d6a5e1-kube-api-access-z5mkc\") pod \"maas-api-revoke-sub-8094b640-xmf6p\" (UID: \"3149dd11-8250-4113-8498-88d836d6a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:30.846134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.845975 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8094b640-xmf6p\" (UID: \"3149dd11-8250-4113-8498-88d836d6a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:30.846176 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.846136 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8094b640-xmf6p\" (UID: \"3149dd11-8250-4113-8498-88d836d6a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:30.946764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.946732 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mkc\" (UniqueName: \"kubernetes.io/projected/3149dd11-8250-4113-8498-88d836d6a5e1-kube-api-access-z5mkc\") pod \"maas-api-revoke-sub-8094b640-xmf6p\" (UID: \"3149dd11-8250-4113-8498-88d836d6a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:30.946941 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.946767 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8094b640-xmf6p\" (UID: \"3149dd11-8250-4113-8498-88d836d6a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:30.947002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.946960 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8094b640-xmf6p\" (UID: \"3149dd11-8250-4113-8498-88d836d6a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:30.953443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.947862 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8094b640-xmf6p\" (UID: \"3149dd11-8250-4113-8498-88d836d6a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:30.953443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.950147 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8094b640-xmf6p\" (UID: \"3149dd11-8250-4113-8498-88d836d6a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:30.970732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.970710 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mkc\" (UniqueName: \"kubernetes.io/projected/3149dd11-8250-4113-8498-88d836d6a5e1-kube-api-access-z5mkc\") pod \"maas-api-revoke-sub-8094b640-xmf6p\" (UID: \"3149dd11-8250-4113-8498-88d836d6a5e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:30.981516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:30.981493 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:31.282048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:31.282020 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p"] Oct 30 04:23:31.282730 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:23:31.282704 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3149dd11_8250_4113_8498_88d836d6a5e1.slice/crio-f34f6b65030902870062b728a5ba6bfc94910e2f24a231377f6fd28910bbd924 WatchSource:0}: Error finding container f34f6b65030902870062b728a5ba6bfc94910e2f24a231377f6fd28910bbd924: Status 404 returned error can't find the container with id f34f6b65030902870062b728a5ba6bfc94910e2f24a231377f6fd28910bbd924 Oct 30 04:23:31.890526 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:31.890444 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:32.032956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:32.032918 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" event={"ID":"3149dd11-8250-4113-8498-88d836d6a5e1","Type":"ContainerStarted","Data":"dc26c49d90ecf6c099ffa418c228a1009d766fe5c748e9d9444321b622cc0a27"} Oct 30 04:23:32.033250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:32.032980 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" event={"ID":"3149dd11-8250-4113-8498-88d836d6a5e1","Type":"ContainerStarted","Data":"f34f6b65030902870062b728a5ba6bfc94910e2f24a231377f6fd28910bbd924"} Oct 30 04:23:32.075519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:32.075463 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" podStartSLOduration=2.075448109 podStartE2EDuration="2.075448109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:23:32.07264158 +0000 UTC m=+1535.358516356" watchObservedRunningTime="2026-10-30 04:23:32.075448109 +0000 UTC m=+1535.361322885" Oct 30 04:23:32.108285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:32.108251 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:33.041307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:33.041276 4793 generic.go:356] "Generic (PLEG): container finished" podID="3149dd11-8250-4113-8498-88d836d6a5e1" containerID="dc26c49d90ecf6c099ffa418c228a1009d766fe5c748e9d9444321b622cc0a27" exitCode=0 Oct 30 04:23:33.041697 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:33.041315 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" event={"ID":"3149dd11-8250-4113-8498-88d836d6a5e1","Type":"ContainerDied","Data":"dc26c49d90ecf6c099ffa418c228a1009d766fe5c748e9d9444321b622cc0a27"} Oct 30 04:23:33.726433 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:33.726386 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:34.212592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:34.212565 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:34.296119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:34.296079 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3149dd11-8250-4113-8498-88d836d6a5e1-kube-api-access-z5mkc\" (UniqueName: \"kubernetes.io/projected/3149dd11-8250-4113-8498-88d836d6a5e1-kube-api-access-z5mkc\") pod \"3149dd11-8250-4113-8498-88d836d6a5e1\" (UID: \"3149dd11-8250-4113-8498-88d836d6a5e1\") " Oct 30 04:23:34.296119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:34.296124 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-openshift-ca\") pod \"3149dd11-8250-4113-8498-88d836d6a5e1\" (UID: \"3149dd11-8250-4113-8498-88d836d6a5e1\") " Oct 30 04:23:34.296364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:34.296183 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-xks-ca\") pod \"3149dd11-8250-4113-8498-88d836d6a5e1\" (UID: \"3149dd11-8250-4113-8498-88d836d6a5e1\") " Oct 30 04:23:34.296653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:34.296620 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-openshift-ca" pod "3149dd11-8250-4113-8498-88d836d6a5e1" (UID: "3149dd11-8250-4113-8498-88d836d6a5e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:23:34.296749 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:34.296721 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:23:34.298520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:34.298368 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3149dd11-8250-4113-8498-88d836d6a5e1-kube-api-access-z5mkc" pod "3149dd11-8250-4113-8498-88d836d6a5e1" (UID: "3149dd11-8250-4113-8498-88d836d6a5e1"). InnerVolumeSpecName "kube-api-access-z5mkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:23:34.298878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:34.298854 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-xks-ca" pod "3149dd11-8250-4113-8498-88d836d6a5e1" (UID: "3149dd11-8250-4113-8498-88d836d6a5e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:23:34.397773 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:34.397746 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5mkc\" (UniqueName: \"kubernetes.io/projected/3149dd11-8250-4113-8498-88d836d6a5e1-kube-api-access-z5mkc\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:23:34.397773 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:34.397774 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3149dd11-8250-4113-8498-88d836d6a5e1-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:23:35.055751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:35.055723 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" Oct 30 04:23:35.055751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:35.055749 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p" event={"ID":"3149dd11-8250-4113-8498-88d836d6a5e1","Type":"ContainerDied","Data":"f34f6b65030902870062b728a5ba6bfc94910e2f24a231377f6fd28910bbd924"} Oct 30 04:23:35.055970 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:35.055805 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f34f6b65030902870062b728a5ba6bfc94910e2f24a231377f6fd28910bbd924" Oct 30 04:23:35.624067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:35.624036 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:35.985617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:35.985592 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" Oct 30 04:23:37.343404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:37.343364 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:39.017707 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:39.017648 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:40.879443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:40.879284 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:41.205441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:41.200113 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:41.232957 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:23:41.232919 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:23:41.432335 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:41.432294 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:41.663068 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:41.662982 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:41.884955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:41.884923 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:42.765958 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:42.765922 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:44.764343 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:44.764308 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:45.099442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:45.099337 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:46.525577 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.525536 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:46.598016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.597982 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx"] Oct 30 04:23:46.598851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.598831 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3149dd11-8250-4113-8498-88d836d6a5e1" containerName="revoke-keys" Oct 30 04:23:46.598905 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.598855 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3149dd11-8250-4113-8498-88d836d6a5e1" containerName="revoke-keys" Oct 30 04:23:46.598992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.598980 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3149dd11-8250-4113-8498-88d836d6a5e1" containerName="revoke-keys" Oct 30 04:23:46.603291 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.603264 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" Oct 30 04:23:46.621365 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.621341 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx"] Oct 30 04:23:46.629931 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.629909 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e124dc\"" Oct 30 04:23:46.629931 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.629914 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e124dc-dockercfg-q8p7h\"" Oct 30 04:23:46.705032 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.704989 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c40414-822f-4a98-ba77-73c7ca563dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx\" (UID: \"74c40414-822f-4a98-ba77-73c7ca563dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" Oct 30 04:23:46.705447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.705404 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbglh\" (UniqueName: \"kubernetes.io/projected/74c40414-822f-4a98-ba77-73c7ca563dd5-kube-api-access-bbglh\") pod \"payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx\" (UID: \"74c40414-822f-4a98-ba77-73c7ca563dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" Oct 30 04:23:46.708131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.708096 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48"] Oct 30 04:23:46.715082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.714397 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" Oct 30 04:23:46.752014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.751982 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:46.807445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.806620 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c40414-822f-4a98-ba77-73c7ca563dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx\" (UID: \"74c40414-822f-4a98-ba77-73c7ca563dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" Oct 30 04:23:46.807445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.806741 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbglh\" (UniqueName: \"kubernetes.io/projected/74c40414-822f-4a98-ba77-73c7ca563dd5-kube-api-access-bbglh\") pod \"payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx\" (UID: \"74c40414-822f-4a98-ba77-73c7ca563dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" Oct 30 04:23:46.807445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.806871 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f1d248-2b4d-4686-a349-04bd92ee3620-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48\" (UID: \"10f1d248-2b4d-4686-a349-04bd92ee3620\") " pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" Oct 30 04:23:46.807445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.806906 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72rn\" (UniqueName: \"kubernetes.io/projected/10f1d248-2b4d-4686-a349-04bd92ee3620-kube-api-access-c72rn\") pod \"payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48\" (UID: \"10f1d248-2b4d-4686-a349-04bd92ee3620\") " pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" Oct 30 04:23:46.813462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.808643 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c40414-822f-4a98-ba77-73c7ca563dd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx\" (UID: \"74c40414-822f-4a98-ba77-73c7ca563dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" Oct 30 04:23:46.816438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.815016 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48"] Oct 30 04:23:46.832458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.831078 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbglh\" (UniqueName: \"kubernetes.io/projected/74c40414-822f-4a98-ba77-73c7ca563dd5-kube-api-access-bbglh\") pod \"payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx\" (UID: \"74c40414-822f-4a98-ba77-73c7ca563dd5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" Oct 30 04:23:46.908768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.908728 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f1d248-2b4d-4686-a349-04bd92ee3620-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48\" (UID: \"10f1d248-2b4d-4686-a349-04bd92ee3620\") " pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" Oct 30 04:23:46.908961 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.908786 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72rn\" (UniqueName: \"kubernetes.io/projected/10f1d248-2b4d-4686-a349-04bd92ee3620-kube-api-access-c72rn\") pod \"payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48\" (UID: \"10f1d248-2b4d-4686-a349-04bd92ee3620\") " pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" Oct 30 04:23:46.910162 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.910130 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f1d248-2b4d-4686-a349-04bd92ee3620-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48\" (UID: \"10f1d248-2b4d-4686-a349-04bd92ee3620\") " pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" Oct 30 04:23:46.917348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.917299 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" Oct 30 04:23:46.934592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.934568 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72rn\" (UniqueName: \"kubernetes.io/projected/10f1d248-2b4d-4686-a349-04bd92ee3620-kube-api-access-c72rn\") pod \"payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48\" (UID: \"10f1d248-2b4d-4686-a349-04bd92ee3620\") " pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" Oct 30 04:23:46.974523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:46.973978 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:47.042006 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:47.041768 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" Oct 30 04:23:47.248797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:47.248722 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx"] Oct 30 04:23:47.267003 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:23:47.266953 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c40414_822f_4a98_ba77_73c7ca563dd5.slice/crio-6c1abdc78e92589cdfe1f7df31e620b1a64a9d93dc39cc27c2b42dcd0c9c6665 WatchSource:0}: Error finding container 6c1abdc78e92589cdfe1f7df31e620b1a64a9d93dc39cc27c2b42dcd0c9c6665: Status 404 returned error can't find the container with id 6c1abdc78e92589cdfe1f7df31e620b1a64a9d93dc39cc27c2b42dcd0c9c6665 Oct 30 04:23:47.377604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:47.377112 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48"] Oct 30 04:23:47.785639 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:47.785601 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:48.206340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:48.206289 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" event={"ID":"10f1d248-2b4d-4686-a349-04bd92ee3620","Type":"ContainerStarted","Data":"1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb"} Oct 30 04:23:48.206589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:48.206364 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" event={"ID":"10f1d248-2b4d-4686-a349-04bd92ee3620","Type":"ContainerStarted","Data":"b7786bf16a0f3a24955b9f65a6ba1938babf2e4b09a7775d9a4bceaf600c60a9"} Oct 30 04:23:48.206589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:48.206398 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" Oct 30 04:23:48.208819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:48.208788 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" event={"ID":"74c40414-822f-4a98-ba77-73c7ca563dd5","Type":"ContainerStarted","Data":"d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3"} Oct 30 04:23:48.208943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:48.208859 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" Oct 30 04:23:48.208943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:48.208880 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" event={"ID":"74c40414-822f-4a98-ba77-73c7ca563dd5","Type":"ContainerStarted","Data":"6c1abdc78e92589cdfe1f7df31e620b1a64a9d93dc39cc27c2b42dcd0c9c6665"} Oct 30 04:23:48.251398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:48.251333 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" podStartSLOduration=2.251316307 podStartE2EDuration="2.251316307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:23:48.247770044 +0000 UTC m=+1551.533644841" watchObservedRunningTime="2026-10-30 04:23:48.251316307 +0000 UTC m=+1551.537191082" Oct 30 04:23:48.290827 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:48.290767 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" podStartSLOduration=2.290749758 podStartE2EDuration="2.290749758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:23:48.289665509 +0000 UTC m=+1551.575540285" watchObservedRunningTime="2026-10-30 04:23:48.290749758 +0000 UTC m=+1551.576624601" Oct 30 04:23:48.822468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:48.822409 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:49.064283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:49.064245 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:50.225305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:50.225275 4793 generic.go:356] "Generic (PLEG): container finished" podID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerID="c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7" exitCode=0 Oct 30 04:23:50.225697 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:50.225351 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" event={"ID":"0a10528f-fdbe-4c27-8b64-67adff5df547","Type":"ContainerDied","Data":"c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7"} Oct 30 04:23:50.225785 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:50.225767 4793 scope.go:122] "RemoveContainer" containerID="c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7" Oct 30 04:23:50.227350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:50.227305 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:23:50.227496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:50.227396 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:23:50.227496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:50.227466 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:23:50.708457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:50.705016 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:50.944002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:50.943965 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:51.194579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:51.194546 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:51.234930 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:51.234895 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" event={"ID":"0a10528f-fdbe-4c27-8b64-67adff5df547","Type":"ContainerStarted","Data":"a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb"} Oct 30 04:23:51.374444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:51.353690 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:51.374444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:51.369554 4793 patch_prober.go:29] interesting pod/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 04:23:51.374444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:51.369637 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 04:23:52.358442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:52.354517 4793 patch_prober.go:29] interesting pod/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 04:23:52.358442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:52.354576 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 04:23:52.528558 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:52.528519 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:52.961371 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:52.961337 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:53.353265 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:53.353188 4793 patch_prober.go:29] interesting pod/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 04:23:53.353265 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:53.353239 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 04:23:54.305946 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:54.305866 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:54.352899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:54.352865 4793 patch_prober.go:29] interesting pod/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 04:23:54.353057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:54.352938 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 04:23:55.353048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:55.353011 4793 patch_prober.go:29] interesting pod/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Oct 30 04:23:55.353481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:55.353061 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Oct 30 04:23:55.922385 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:55.922345 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:56.363716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:56.363683 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:56.364921 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:56.364894 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:56.369150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:56.369124 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:23:57.041721 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:57.041692 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx"] Oct 30 04:23:57.053411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:57.053383 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef178ec8-2wjvx"] Oct 30 04:23:57.177272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:57.177241 4793 scope.go:122] "RemoveContainer" containerID="a041b1769079cf2907fe477363301aa6cc9321493653580b774dd8a25d00e85e" Oct 30 04:23:57.188942 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:57.188922 4793 scope.go:122] "RemoveContainer" containerID="bb21beeb858af6fc28ab5a6ffd2c61f75bda28892411e903a82ea435f2a4b2c4" Oct 30 04:23:57.198881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:57.198855 4793 scope.go:122] "RemoveContainer" containerID="cf9c8e75be8e6aad1e7c009fe0dd0dc883c6f5528bfef409a85636b3ab6130cd" Oct 30 04:23:57.207162 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:57.207141 4793 scope.go:122] "RemoveContainer" containerID="573d5caaea335baa8583d87694cd5fd29268f2dd92918b89d780741bf0fc7791" Oct 30 04:23:57.217141 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:57.217121 4793 scope.go:122] "RemoveContainer" containerID="262f29933a9df8db0cc4376515785dee3123bd56dc2550a4c916b874e688176a" Oct 30 04:23:57.578988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:57.578948 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:57.796959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:57.796916 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:23:58.848213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:58.848183 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d68363-4dde-45c1-91e0-d9bde42a703a" path="/var/lib/kubelet/pods/b0d68363-4dde-45c1-91e0-d9bde42a703a/volumes" Oct 30 04:23:59.227523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:59.227474 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" Oct 30 04:23:59.238779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:59.238748 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" Oct 30 04:23:59.281107 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:23:59.278144 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:01.026307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:01.026271 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:02.695814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:02.695770 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:04.306127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:04.306087 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:05.190822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.190791 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4"] Oct 30 04:24:05.197091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.197063 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.234495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.234409 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4"] Oct 30 04:24:05.306935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.306885 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.306935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.306933 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.307504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.306958 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.307504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.307104 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhtk\" (UniqueName: \"kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-kube-api-access-wwhtk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.307504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.307149 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.307504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.307250 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.307504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.307310 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.307504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.307332 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.307836 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.307560 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.408861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.408822 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.408867 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.409034 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.409067 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.409093 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.409118 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.409174 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhtk\" (UniqueName: \"kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-kube-api-access-wwhtk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.409217 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.409260 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.409293 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.409179 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.409441 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409715 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.409629 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.409786 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.409774 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.411477 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.411456 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.411699 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.411676 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.432607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.432577 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.432801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.432783 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhtk\" (UniqueName: \"kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-kube-api-access-wwhtk\") pod \"maas-default-gateway-openshift-default-7f694df8bc-5jxq4\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.512079 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.512019 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:05.842685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.842657 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4"] Oct 30 04:24:05.843172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.843112 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:05.843255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.843212 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:05.843255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:05.843245 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:06.064094 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:06.064000 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:06.414641 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:06.414600 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" event={"ID":"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda","Type":"ContainerStarted","Data":"e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df"} Oct 30 04:24:06.415025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:06.414671 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" event={"ID":"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda","Type":"ContainerStarted","Data":"bc480f961b55af67dd8f986305b293de61b6f8e02c9ec724c9b6b49cdd6416cf"} Oct 30 04:24:06.471282 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:06.471225 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" podStartSLOduration=1.471209486 podStartE2EDuration="1.471209486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:06.470629149 +0000 UTC m=+1569.756503927" watchObservedRunningTime="2026-10-30 04:24:06.471209486 +0000 UTC m=+1569.757084261" Oct 30 04:24:06.513178 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:06.513147 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:06.514783 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:06.514759 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5jxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 04:24:06.514916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:06.514803 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 04:24:06.939620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:06.939587 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:07.299107 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.299030 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg"] Oct 30 04:24:07.304276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.304246 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.327487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.327396 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg"] Oct 30 04:24:07.335595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.335564 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e3c43b-openshift-default-dockercfg-clszw\"" Oct 30 04:24:07.464445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.450038 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-certs\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.538847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.490496 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:07.539303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.539280 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-socket\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.544016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.543989 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-podinfo\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.553873 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.553794 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istiod-ca-cert\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.554167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.554145 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-envoy\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.554367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.554346 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-token\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.554567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.554549 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-data\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.554747 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.554730 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-credential-socket\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.554955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.554940 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wsw\" (UniqueName: \"kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-kube-api-access-67wsw\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.571218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.571185 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5jxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 04:24:07.571484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.571456 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 04:24:07.672444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.668219 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-token\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.672444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.668286 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-data\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.672444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.668321 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-credential-socket\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.672444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.668408 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wsw\" (UniqueName: \"kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-kube-api-access-67wsw\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.672444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.668546 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-certs\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.672444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.668660 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-socket\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.672444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.668701 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-podinfo\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.672444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.668752 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istiod-ca-cert\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.672444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.668804 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-envoy\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.690450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.688475 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-data\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.690450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.688730 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-credential-socket\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.690450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.689384 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-certs\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.691376 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.691352 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istiod-ca-cert\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.692757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.691770 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-socket\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.697442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.694697 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-envoy\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.703685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.703137 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-podinfo\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.714001 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.713939 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-token\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.716542 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.716517 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wsw\" (UniqueName: \"kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-kube-api-access-67wsw\") pod \"e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:07.781397 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.781309 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:07.923133 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:07.923041 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:08.008500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:08.008465 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:08.294594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:08.294557 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg"] Oct 30 04:24:08.295573 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:24:08.295545 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d17eab8_a71b_4fc5_b66b_e8026f8f874c.slice/crio-7342b6fd453698195595d95e3c0d866c1986c9774dbf5f44775bee23895fefab WatchSource:0}: Error finding container 7342b6fd453698195595d95e3c0d866c1986c9774dbf5f44775bee23895fefab: Status 404 returned error can't find the container with id 7342b6fd453698195595d95e3c0d866c1986c9774dbf5f44775bee23895fefab Oct 30 04:24:08.298288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:08.298254 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:08.298364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:08.298320 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:08.298364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:08.298351 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:08.511310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:08.510811 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" event={"ID":"1d17eab8-a71b-4fc5-b66b-e8026f8f874c","Type":"ContainerStarted","Data":"9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae"} Oct 30 04:24:08.511310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:08.510871 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" event={"ID":"1d17eab8-a71b-4fc5-b66b-e8026f8f874c","Type":"ContainerStarted","Data":"7342b6fd453698195595d95e3c0d866c1986c9774dbf5f44775bee23895fefab"} Oct 30 04:24:08.512517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:08.512492 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5jxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 04:24:08.512600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:08.512543 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 04:24:08.596492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:08.596110 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" podStartSLOduration=1.596088778 podStartE2EDuration="1.596088778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:08.569190927 +0000 UTC m=+1571.855065739" watchObservedRunningTime="2026-10-30 04:24:08.596088778 +0000 UTC m=+1571.881963558" Oct 30 04:24:08.924215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:08.924134 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:09.513014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:09.512928 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5jxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 04:24:09.513453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:09.513005 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 04:24:09.690142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:09.690106 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:09.929659 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:09.929634 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:10.513160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:10.513122 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5jxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 04:24:10.513649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:10.513192 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 04:24:10.525011 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:10.524979 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:10.525831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:10.525812 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:11.105440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:11.105376 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:11.220944 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:24:11.220909 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:24:11.415500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:11.415298 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:11.513206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:11.513166 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5jxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 04:24:11.513594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:11.513248 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 04:24:11.682240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:11.682197 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:12.512311 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:12.512277 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5jxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 04:24:12.512495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:12.512345 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 04:24:12.849722 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:12.849636 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:12.925234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:12.925202 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4"] Oct 30 04:24:12.928938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:12.928921 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" Oct 30 04:24:12.946507 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:12.946482 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4"] Oct 30 04:24:12.961901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:12.961874 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e3c43b\"" Oct 30 04:24:12.962044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:12.961970 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e3c43b-dockercfg-p8tqp\"" Oct 30 04:24:13.001075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.001030 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r8zc\" (UniqueName: \"kubernetes.io/projected/31400a77-11ef-4706-999a-5f60cee5ecf7-kube-api-access-9r8zc\") pod \"payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4\" (UID: \"31400a77-11ef-4706-999a-5f60cee5ecf7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" Oct 30 04:24:13.001267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.001238 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31400a77-11ef-4706-999a-5f60cee5ecf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4\" (UID: \"31400a77-11ef-4706-999a-5f60cee5ecf7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" Oct 30 04:24:13.091443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.091380 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:13.119663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.117954 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r8zc\" (UniqueName: \"kubernetes.io/projected/31400a77-11ef-4706-999a-5f60cee5ecf7-kube-api-access-9r8zc\") pod \"payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4\" (UID: \"31400a77-11ef-4706-999a-5f60cee5ecf7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" Oct 30 04:24:13.119663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.118146 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31400a77-11ef-4706-999a-5f60cee5ecf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4\" (UID: \"31400a77-11ef-4706-999a-5f60cee5ecf7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" Oct 30 04:24:13.119663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.119125 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31400a77-11ef-4706-999a-5f60cee5ecf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4\" (UID: \"31400a77-11ef-4706-999a-5f60cee5ecf7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" Oct 30 04:24:13.128481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.128456 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq"] Oct 30 04:24:13.139589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.139550 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f09c077-cg9sq"] Oct 30 04:24:13.158834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.158787 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r8zc\" (UniqueName: \"kubernetes.io/projected/31400a77-11ef-4706-999a-5f60cee5ecf7-kube-api-access-9r8zc\") pod \"payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4\" (UID: \"31400a77-11ef-4706-999a-5f60cee5ecf7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" Oct 30 04:24:13.247972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.247938 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" Oct 30 04:24:13.513332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.513299 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-5jxq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 04:24:13.513524 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.513367 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 04:24:13.605400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.605367 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:13.635005 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:24:13.634946 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31400a77_11ef_4706_999a_5f60cee5ecf7.slice/crio-ced072ad42d20d0bce3dfbafc82e0eec30ed368924a45773138aa1ff85abf627 WatchSource:0}: Error finding container ced072ad42d20d0bce3dfbafc82e0eec30ed368924a45773138aa1ff85abf627: Status 404 returned error can't find the container with id ced072ad42d20d0bce3dfbafc82e0eec30ed368924a45773138aa1ff85abf627 Oct 30 04:24:13.635552 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:13.635528 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4"] Oct 30 04:24:14.567371 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:14.567326 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" event={"ID":"31400a77-11ef-4706-999a-5f60cee5ecf7","Type":"ContainerStarted","Data":"9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be"} Oct 30 04:24:14.567921 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:14.567395 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" event={"ID":"31400a77-11ef-4706-999a-5f60cee5ecf7","Type":"ContainerStarted","Data":"ced072ad42d20d0bce3dfbafc82e0eec30ed368924a45773138aa1ff85abf627"} Oct 30 04:24:14.568383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:14.568358 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" Oct 30 04:24:14.616062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:14.614444 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" podStartSLOduration=2.61441035 podStartE2EDuration="2.61441035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:14.612856746 +0000 UTC m=+1577.898731523" watchObservedRunningTime="2026-10-30 04:24:14.61441035 +0000 UTC m=+1577.900285125" Oct 30 04:24:14.852006 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:14.851923 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fe9fdc-4b19-494e-8c29-455263ac88f7" path="/var/lib/kubelet/pods/c0fe9fdc-4b19-494e-8c29-455263ac88f7/volumes" Oct 30 04:24:14.990905 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:14.990869 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:14.991788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:14.991685 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:14.992680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:14.992632 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:24:15.159229 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:15.159144 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:15.544776 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:15.544721 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:15.864460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:15.864362 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:16.358808 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:16.358721 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:16.579671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:16.579629 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:17.204872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:17.204834 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:19.026006 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:19.025967 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:20.052032 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.051975 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:20.238451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.238398 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk"] Oct 30 04:24:20.242619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.242598 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.275924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.275741 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk"] Oct 30 04:24:20.292076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.292046 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:20.342742 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.342671 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm4mg\" (UniqueName: \"kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-kube-api-access-jm4mg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.342742 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.342734 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70316b40-a67f-477f-886a-96779807d046-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.342926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.342835 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70316b40-a67f-477f-886a-96779807d046-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.342926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.342899 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.343025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.343004 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.343083 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.343065 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.343269 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.343244 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.343318 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.343298 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.343374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.343357 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.444934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.444901 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.445080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.444967 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.445080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.445069 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.445159 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.445103 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.445159 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.445132 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.445240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.445173 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm4mg\" (UniqueName: \"kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-kube-api-access-jm4mg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.445240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.445229 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70316b40-a67f-477f-886a-96779807d046-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.445349 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.445284 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70316b40-a67f-477f-886a-96779807d046-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.445404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.445350 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.445496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.445475 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.445551 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.445517 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.445620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.445560 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.445858 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.445812 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.446136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.446111 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70316b40-a67f-477f-886a-96779807d046-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.447636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.447612 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.447903 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.447882 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70316b40-a67f-477f-886a-96779807d046-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.478571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.478546 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.478863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.478842 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm4mg\" (UniqueName: \"kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-kube-api-access-jm4mg\") pod \"maas-default-gateway-openshift-default-7f694df8bc-78gqk\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.557774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.557747 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:20.878248 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:24:20.878218 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70316b40_a67f_477f_886a_96779807d046.slice/crio-96da11553e15f1f4f7d256a03ede9525743f42e95ecc9525f7ca0e9e50d3742a WatchSource:0}: Error finding container 96da11553e15f1f4f7d256a03ede9525743f42e95ecc9525f7ca0e9e50d3742a: Status 404 returned error can't find the container with id 96da11553e15f1f4f7d256a03ede9525743f42e95ecc9525f7ca0e9e50d3742a Oct 30 04:24:20.878380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.878349 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk"] Oct 30 04:24:20.881075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.881039 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:20.881195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.881115 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:20.881195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:20.881162 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:21.000217 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.000169 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:21.622901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.622861 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" event={"ID":"70316b40-a67f-477f-886a-96779807d046","Type":"ContainerStarted","Data":"282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca"} Oct 30 04:24:21.623403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.622923 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" event={"ID":"70316b40-a67f-477f-886a-96779807d046","Type":"ContainerStarted","Data":"96da11553e15f1f4f7d256a03ede9525743f42e95ecc9525f7ca0e9e50d3742a"} Oct 30 04:24:21.697342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.697264 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" podStartSLOduration=1.697242218 podStartE2EDuration="1.697242218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:21.683969259 +0000 UTC m=+1584.969844059" watchObservedRunningTime="2026-10-30 04:24:21.697242218 +0000 UTC m=+1584.983116994" Oct 30 04:24:21.721878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.721846 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:21.792793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.792758 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz"] Oct 30 04:24:21.797829 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.797804 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" Oct 30 04:24:21.811578 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.811556 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz"] Oct 30 04:24:21.812937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.812916 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0c530e\"" Oct 30 04:24:21.813063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.812991 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0c530e-dockercfg-zrntk\"" Oct 30 04:24:21.951441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.943208 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:21.951441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.948512 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj"] Oct 30 04:24:21.963447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.961885 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" Oct 30 04:24:21.987748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.981928 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cblf\" (UniqueName: \"kubernetes.io/projected/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-kube-api-access-5cblf\") pod \"payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz\" (UID: \"1d74fd83-dd2d-4b3e-8269-b96ff52ed274\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" Oct 30 04:24:21.987748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:21.982057 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz\" (UID: \"1d74fd83-dd2d-4b3e-8269-b96ff52ed274\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" Oct 30 04:24:22.115531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.115479 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj"] Oct 30 04:24:22.123597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.123554 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cblf\" (UniqueName: \"kubernetes.io/projected/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-kube-api-access-5cblf\") pod \"payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz\" (UID: \"1d74fd83-dd2d-4b3e-8269-b96ff52ed274\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" Oct 30 04:24:22.123777 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.123636 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98d8m\" (UniqueName: \"kubernetes.io/projected/025f0ebe-c1e2-439f-8072-8543c281269a-kube-api-access-98d8m\") pod \"payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj\" (UID: \"025f0ebe-c1e2-439f-8072-8543c281269a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" Oct 30 04:24:22.123777 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.123750 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz\" (UID: \"1d74fd83-dd2d-4b3e-8269-b96ff52ed274\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" Oct 30 04:24:22.123893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.123791 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025f0ebe-c1e2-439f-8072-8543c281269a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj\" (UID: \"025f0ebe-c1e2-439f-8072-8543c281269a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" Oct 30 04:24:22.125069 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.124934 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz\" (UID: \"1d74fd83-dd2d-4b3e-8269-b96ff52ed274\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" Oct 30 04:24:22.147175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.147152 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cblf\" (UniqueName: \"kubernetes.io/projected/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-kube-api-access-5cblf\") pod \"payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz\" (UID: \"1d74fd83-dd2d-4b3e-8269-b96ff52ed274\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" Oct 30 04:24:22.225411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.225323 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98d8m\" (UniqueName: \"kubernetes.io/projected/025f0ebe-c1e2-439f-8072-8543c281269a-kube-api-access-98d8m\") pod \"payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj\" (UID: \"025f0ebe-c1e2-439f-8072-8543c281269a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" Oct 30 04:24:22.225411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.225404 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025f0ebe-c1e2-439f-8072-8543c281269a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj\" (UID: \"025f0ebe-c1e2-439f-8072-8543c281269a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" Oct 30 04:24:22.226220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.226189 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025f0ebe-c1e2-439f-8072-8543c281269a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj\" (UID: \"025f0ebe-c1e2-439f-8072-8543c281269a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" Oct 30 04:24:22.249274 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.249240 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98d8m\" (UniqueName: \"kubernetes.io/projected/025f0ebe-c1e2-439f-8072-8543c281269a-kube-api-access-98d8m\") pod \"payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj\" (UID: \"025f0ebe-c1e2-439f-8072-8543c281269a\") " pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" Oct 30 04:24:22.294874 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.294831 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" Oct 30 04:24:22.415158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.415126 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" Oct 30 04:24:22.595969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.594869 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:22.600384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.600321 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78gqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 04:24:22.600802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.600444 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" podUID="70316b40-a67f-477f-886a-96779807d046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 04:24:22.637407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.637375 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj"] Oct 30 04:24:22.637851 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:24:22.637376 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025f0ebe_c1e2_439f_8072_8543c281269a.slice/crio-0674fc348e3f6a35c81dd41cc81ed878227608ca31a5b57992015e33dd7d9f58 WatchSource:0}: Error finding container 0674fc348e3f6a35c81dd41cc81ed878227608ca31a5b57992015e33dd7d9f58: Status 404 returned error can't find the container with id 0674fc348e3f6a35c81dd41cc81ed878227608ca31a5b57992015e33dd7d9f58 Oct 30 04:24:22.809512 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.808941 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz"] Oct 30 04:24:22.814450 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:24:22.811953 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d74fd83_dd2d_4b3e_8269_b96ff52ed274.slice/crio-bbd45ea8f73b29771a8db119d2e9b75e61942cb3255ecf3dff495e5b8d72b996 WatchSource:0}: Error finding container bbd45ea8f73b29771a8db119d2e9b75e61942cb3255ecf3dff495e5b8d72b996: Status 404 returned error can't find the container with id bbd45ea8f73b29771a8db119d2e9b75e61942cb3255ecf3dff495e5b8d72b996 Oct 30 04:24:22.821477 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:22.821168 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:23.562444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:23.558750 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78gqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 04:24:23.562444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:23.558815 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" podUID="70316b40-a67f-477f-886a-96779807d046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 04:24:23.644256 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:23.644223 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" event={"ID":"1d74fd83-dd2d-4b3e-8269-b96ff52ed274","Type":"ContainerStarted","Data":"90d70676903950852c71bebf9115a25bd5f8ed3f5380a797e6734a2ac04ddbb1"} Oct 30 04:24:23.644765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:23.644278 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" event={"ID":"1d74fd83-dd2d-4b3e-8269-b96ff52ed274","Type":"ContainerStarted","Data":"bbd45ea8f73b29771a8db119d2e9b75e61942cb3255ecf3dff495e5b8d72b996"} Oct 30 04:24:23.644765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:23.644451 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" Oct 30 04:24:23.646161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:23.646134 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" event={"ID":"025f0ebe-c1e2-439f-8072-8543c281269a","Type":"ContainerStarted","Data":"9bd8174456efa9ac08c81d4dd77aa0565e4146a168c1a4d60079c4a0c4d42221"} Oct 30 04:24:23.646291 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:23.646185 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" event={"ID":"025f0ebe-c1e2-439f-8072-8543c281269a","Type":"ContainerStarted","Data":"0674fc348e3f6a35c81dd41cc81ed878227608ca31a5b57992015e33dd7d9f58"} Oct 30 04:24:23.646368 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:23.646352 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" Oct 30 04:24:23.701662 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:23.701611 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" podStartSLOduration=2.701598769 podStartE2EDuration="2.701598769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:23.69956375 +0000 UTC m=+1586.985438546" watchObservedRunningTime="2026-10-30 04:24:23.701598769 +0000 UTC m=+1586.987473576" Oct 30 04:24:23.741793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:23.741741 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" podStartSLOduration=2.741728629 podStartE2EDuration="2.741728629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:23.741016562 +0000 UTC m=+1587.026891362" watchObservedRunningTime="2026-10-30 04:24:23.741728629 +0000 UTC m=+1587.027603404" Oct 30 04:24:24.588444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:24.567929 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78gqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 04:24:24.588444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:24.567989 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" podUID="70316b40-a67f-477f-886a-96779807d046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 04:24:24.619395 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:24.618150 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:25.032629 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:25.032592 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz"] Oct 30 04:24:25.042739 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:25.042713 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7090644b-f4qsz"] Oct 30 04:24:25.574935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:25.574506 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78gqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 04:24:25.574935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:25.574569 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" podUID="70316b40-a67f-477f-886a-96779807d046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 04:24:26.398404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:26.398362 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:26.559441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:26.558724 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78gqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 04:24:26.559441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:26.558780 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" podUID="70316b40-a67f-477f-886a-96779807d046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 04:24:26.587657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:26.587627 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" Oct 30 04:24:26.623496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:26.623471 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:26.847985 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:26.847956 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8be392d-2a0b-4813-837a-0492164756fe" path="/var/lib/kubelet/pods/a8be392d-2a0b-4813-837a-0492164756fe/volumes" Oct 30 04:24:27.558674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:27.558633 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78gqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 04:24:27.559143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:27.558684 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" podUID="70316b40-a67f-477f-886a-96779807d046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 04:24:28.030405 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:28.030372 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc"] Oct 30 04:24:28.042477 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:28.042448 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a7cd0f9-kcgpc"] Oct 30 04:24:28.122161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:28.122126 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:28.558753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:28.558714 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78gqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 04:24:28.559136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:28.558780 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" podUID="70316b40-a67f-477f-886a-96779807d046" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 04:24:28.848592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:28.848513 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb3b71f0-6a23-4da7-b457-f53a13fb105e" path="/var/lib/kubelet/pods/eb3b71f0-6a23-4da7-b457-f53a13fb105e/volumes" Oct 30 04:24:29.032609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:29.032581 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6"] Oct 30 04:24:29.044241 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:29.044214 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a214362-xvpr6"] Oct 30 04:24:29.733739 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:29.733711 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:29.734185 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:29.734113 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:29.735867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:29.735842 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:24:29.821260 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:29.821226 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:30.847647 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:30.847616 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09544a9a-3793-4039-aa9a-821a4e1aeaa0" path="/var/lib/kubelet/pods/09544a9a-3793-4039-aa9a-821a4e1aeaa0/volumes" Oct 30 04:24:31.580586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:31.580552 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:32.446265 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:32.446220 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:33.393136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:33.393101 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:34.666841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:34.666811 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" Oct 30 04:24:34.667269 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:34.666893 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" Oct 30 04:24:35.064846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:35.064809 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:35.283345 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:35.283304 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:36.921085 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:36.921050 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:38.515062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:38.514989 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:40.184452 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:40.184385 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:41.219101 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:24:41.219068 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:24:41.994492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:41.992541 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:43.602442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:43.602361 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:45.214075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:45.214044 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:46.030308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:46.030274 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch"] Oct 30 04:24:46.042278 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:46.042254 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8572a84-zjxch"] Oct 30 04:24:46.630971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:46.630931 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:46.847798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:46.847764 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417dfae5-0ce8-414c-b0d4-86321d544654" path="/var/lib/kubelet/pods/417dfae5-0ce8-414c-b0d4-86321d544654/volumes" Oct 30 04:24:46.991929 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:46.991887 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:47.361797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.361698 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:47.756546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.756493 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5"] Oct 30 04:24:47.764301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.764268 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.779627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.779566 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-397bda-openshift-default-dockercfg-g4gz4\"" Oct 30 04:24:47.786952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.786917 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5"] Oct 30 04:24:47.836222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.836188 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e56157-cea5-4efb-86a3-797f64913b05-istio-podinfo\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.836222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.836223 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-istio-token\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.836485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.836326 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-credential-socket\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.836485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.836359 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e56157-cea5-4efb-86a3-797f64913b05-istiod-ca-cert\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.836594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.836532 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-data\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.836646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.836626 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmqsc\" (UniqueName: \"kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-kube-api-access-fmqsc\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.836701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.836688 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-envoy\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.836782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.836763 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-certs\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.836840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.836801 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-socket\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.927163 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.927050 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:47.940831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.938124 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-data\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.940831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.938260 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmqsc\" (UniqueName: \"kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-kube-api-access-fmqsc\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.940831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.938365 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-envoy\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.940831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.938473 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-certs\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.940831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.938522 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-socket\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.940831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.938563 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e56157-cea5-4efb-86a3-797f64913b05-istio-podinfo\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.940831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.938589 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-istio-token\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.940831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.938691 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-credential-socket\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.940831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.938753 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e56157-cea5-4efb-86a3-797f64913b05-istiod-ca-cert\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.940831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.939571 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-data\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.943965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.942249 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-certs\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.943965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.942837 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-socket\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.943965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.943120 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-credential-socket\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.943965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.943868 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e56157-cea5-4efb-86a3-797f64913b05-istiod-ca-cert\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.963269 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.963193 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e56157-cea5-4efb-86a3-797f64913b05-istio-podinfo\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:47.984044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:47.982171 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-envoy\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:48.015493 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:48.005996 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-istio-token\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:48.017270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:48.017184 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmqsc\" (UniqueName: \"kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-kube-api-access-fmqsc\") pod \"e2e-api-b-397bda-openshift-default-54899c56f9-mddj5\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:48.087936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:48.087722 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:48.128370 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:48.128325 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:48.409550 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:24:48.409522 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e56157_cea5_4efb_86a3_797f64913b05.slice/crio-0185f951b2a299aaa6ad01a64fa5faaec2c41970410b298e079d528939e50ba7 WatchSource:0}: Error finding container 0185f951b2a299aaa6ad01a64fa5faaec2c41970410b298e079d528939e50ba7: Status 404 returned error can't find the container with id 0185f951b2a299aaa6ad01a64fa5faaec2c41970410b298e079d528939e50ba7 Oct 30 04:24:48.411941 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:48.411916 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5"] Oct 30 04:24:48.412142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:48.412109 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:48.412184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:48.412174 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:48.412225 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:48.412201 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:48.714211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:48.714173 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:49.023172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:49.023096 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" event={"ID":"e5e56157-cea5-4efb-86a3-797f64913b05","Type":"ContainerStarted","Data":"3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7"} Oct 30 04:24:49.023172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:49.023151 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" event={"ID":"e5e56157-cea5-4efb-86a3-797f64913b05","Type":"ContainerStarted","Data":"0185f951b2a299aaa6ad01a64fa5faaec2c41970410b298e079d528939e50ba7"} Oct 30 04:24:49.084721 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:49.084668 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" podStartSLOduration=2.084653084 podStartE2EDuration="2.084653084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:49.081391522 +0000 UTC m=+1612.367266332" watchObservedRunningTime="2026-10-30 04:24:49.084653084 +0000 UTC m=+1612.370527858" Oct 30 04:24:49.088393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:49.088365 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:49.096445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:49.096399 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:50.032033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:50.032001 4793 generic.go:356] "Generic (PLEG): container finished" podID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerID="9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae" exitCode=0 Oct 30 04:24:50.032472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:50.032073 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" event={"ID":"1d17eab8-a71b-4fc5-b66b-e8026f8f874c","Type":"ContainerDied","Data":"9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae"} Oct 30 04:24:50.032472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:50.032371 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:50.032709 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:50.032692 4793 scope.go:122] "RemoveContainer" containerID="9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae" Oct 30 04:24:50.033632 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:50.033261 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:24:50.034091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:50.034058 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:50.034191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:50.034140 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:50.034253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:50.034188 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:24:50.342455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:50.342353 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:51.042285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:51.042254 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" event={"ID":"1d17eab8-a71b-4fc5-b66b-e8026f8f874c","Type":"ContainerStarted","Data":"d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff"} Oct 30 04:24:51.670444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:51.670116 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:51.879884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:51.879849 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:51.924290 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:51.924211 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:51.925357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:51.925324 4793 patch_prober.go:29] interesting pod/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 04:24:51.925515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:51.925390 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 04:24:52.302440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:52.302333 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:52.924320 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:52.924278 4793 patch_prober.go:29] interesting pod/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 04:24:52.924523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:52.924357 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 04:24:53.072919 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.072882 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:53.384608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.384567 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:53.489383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.489341 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm"] Oct 30 04:24:53.494934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.494914 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" Oct 30 04:24:53.511565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.511541 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm"] Oct 30 04:24:53.520045 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.520016 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-397bda-dockercfg-tbzxm\"" Oct 30 04:24:53.560443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.546715 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-397bda\"" Oct 30 04:24:53.636559 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.626966 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:53.646473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.644641 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d55806-458c-454d-998f-f56e3eb4363f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm\" (UID: \"c4d55806-458c-454d-998f-f56e3eb4363f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" Oct 30 04:24:53.646473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.644743 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qml8t\" (UniqueName: \"kubernetes.io/projected/c4d55806-458c-454d-998f-f56e3eb4363f-kube-api-access-qml8t\") pod \"payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm\" (UID: \"c4d55806-458c-454d-998f-f56e3eb4363f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" Oct 30 04:24:53.746082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.746037 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d55806-458c-454d-998f-f56e3eb4363f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm\" (UID: \"c4d55806-458c-454d-998f-f56e3eb4363f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" Oct 30 04:24:53.746339 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.746152 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qml8t\" (UniqueName: \"kubernetes.io/projected/c4d55806-458c-454d-998f-f56e3eb4363f-kube-api-access-qml8t\") pod \"payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm\" (UID: \"c4d55806-458c-454d-998f-f56e3eb4363f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" Oct 30 04:24:53.747605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.747565 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d55806-458c-454d-998f-f56e3eb4363f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm\" (UID: \"c4d55806-458c-454d-998f-f56e3eb4363f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" Oct 30 04:24:53.807442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.806554 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qml8t\" (UniqueName: \"kubernetes.io/projected/c4d55806-458c-454d-998f-f56e3eb4363f-kube-api-access-qml8t\") pod \"payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm\" (UID: \"c4d55806-458c-454d-998f-f56e3eb4363f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" Oct 30 04:24:53.817713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.816092 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" Oct 30 04:24:53.931443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.927785 4793 patch_prober.go:29] interesting pod/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 04:24:53.931443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:53.927844 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 04:24:54.220521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:54.219862 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:54.318921 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:24:54.318891 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d55806_458c_454d_998f_f56e3eb4363f.slice/crio-85c2be508a67295361dc570416e7f4855ff2126c02f4864e82f15f64e46d7359 WatchSource:0}: Error finding container 85c2be508a67295361dc570416e7f4855ff2126c02f4864e82f15f64e46d7359: Status 404 returned error can't find the container with id 85c2be508a67295361dc570416e7f4855ff2126c02f4864e82f15f64e46d7359 Oct 30 04:24:54.319084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:54.319059 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm"] Oct 30 04:24:54.924166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:54.924129 4793 patch_prober.go:29] interesting pod/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 04:24:54.924681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:54.924202 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 04:24:55.112777 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:55.112737 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" event={"ID":"c4d55806-458c-454d-998f-f56e3eb4363f","Type":"ContainerStarted","Data":"296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1"} Oct 30 04:24:55.112954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:55.112802 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" event={"ID":"c4d55806-458c-454d-998f-f56e3eb4363f","Type":"ContainerStarted","Data":"85c2be508a67295361dc570416e7f4855ff2126c02f4864e82f15f64e46d7359"} Oct 30 04:24:55.112954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:55.112830 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" Oct 30 04:24:55.173441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:55.166270 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" podStartSLOduration=2.16625167 podStartE2EDuration="2.16625167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:24:55.16349003 +0000 UTC m=+1618.449364824" watchObservedRunningTime="2026-10-30 04:24:55.16625167 +0000 UTC m=+1618.452126447" Oct 30 04:24:55.526589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:55.526549 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:55.572442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:55.571981 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:55.924309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:55.924228 4793 patch_prober.go:29] interesting pod/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 04:24:55.924309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:55.924291 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 04:24:56.010438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.004492 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:56.032753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.032722 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr"] Oct 30 04:24:56.044708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.044679 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76dede6a-6jbxr"] Oct 30 04:24:56.541315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.541263 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq"] Oct 30 04:24:56.548057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.548020 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:24:56.564975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.564947 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq"] Oct 30 04:24:56.571944 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.571918 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:24:56.572080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.571955 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:24:56.583953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.583889 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:24:56.584079 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.584061 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:24:56.656116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.656086 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq\" (UID: \"371744d4-b7e1-4021-8f03-3bc955b6aab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:24:56.656245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.656122 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq\" (UID: \"371744d4-b7e1-4021-8f03-3bc955b6aab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:24:56.656328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.656279 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5trv\" (UniqueName: \"kubernetes.io/projected/371744d4-b7e1-4021-8f03-3bc955b6aab3-kube-api-access-p5trv\") pod \"maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq\" (UID: \"371744d4-b7e1-4021-8f03-3bc955b6aab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:24:56.757195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.757155 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq\" (UID: \"371744d4-b7e1-4021-8f03-3bc955b6aab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:24:56.757367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.757271 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5trv\" (UniqueName: \"kubernetes.io/projected/371744d4-b7e1-4021-8f03-3bc955b6aab3-kube-api-access-p5trv\") pod \"maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq\" (UID: \"371744d4-b7e1-4021-8f03-3bc955b6aab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:24:56.757530 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.757514 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq\" (UID: \"371744d4-b7e1-4021-8f03-3bc955b6aab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:24:56.758300 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.758277 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq\" (UID: \"371744d4-b7e1-4021-8f03-3bc955b6aab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:24:56.759535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.759503 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq\" (UID: \"371744d4-b7e1-4021-8f03-3bc955b6aab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:24:56.780659 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.780632 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5trv\" (UniqueName: \"kubernetes.io/projected/371744d4-b7e1-4021-8f03-3bc955b6aab3-kube-api-access-p5trv\") pod \"maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq\" (UID: \"371744d4-b7e1-4021-8f03-3bc955b6aab3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:24:56.847876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.847816 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da2b78f6-613a-4163-8ba0-fcfbd227d95a" path="/var/lib/kubelet/pods/da2b78f6-613a-4163-8ba0-fcfbd227d95a/volumes" Oct 30 04:24:56.880596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.880575 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:24:56.886429 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.886400 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:24:56.928788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.928757 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:56.930202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:56.930165 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:24:57.191890 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:24:57.191856 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371744d4_b7e1_4021_8f03_3bc955b6aab3.slice/crio-d43cd679ceaaece6bfda843daddda80d65e66d82d68c3f3fc2975b734f206313 WatchSource:0}: Error finding container d43cd679ceaaece6bfda843daddda80d65e66d82d68c3f3fc2975b734f206313: Status 404 returned error can't find the container with id d43cd679ceaaece6bfda843daddda80d65e66d82d68c3f3fc2975b734f206313 Oct 30 04:24:57.192666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:57.192640 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq"] Oct 30 04:24:57.264646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:57.264622 4793 scope.go:122] "RemoveContainer" containerID="fa4367f605055f431c6a1a79b0fc51225f04f4fb298bc528603529d2bc544356" Oct 30 04:24:57.790694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:57.790511 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:58.137783 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:58.137748 4793 generic.go:356] "Generic (PLEG): container finished" podID="371744d4-b7e1-4021-8f03-3bc955b6aab3" containerID="6a9e12199d1c64b9a16dd0ee9e7585b8826400659e69715e9a3b4ccf2ef5be54" exitCode=0 Oct 30 04:24:58.138184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:58.137831 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" event={"ID":"371744d4-b7e1-4021-8f03-3bc955b6aab3","Type":"ContainerDied","Data":"6a9e12199d1c64b9a16dd0ee9e7585b8826400659e69715e9a3b4ccf2ef5be54"} Oct 30 04:24:58.138184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:58.137892 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" event={"ID":"371744d4-b7e1-4021-8f03-3bc955b6aab3","Type":"ContainerStarted","Data":"d43cd679ceaaece6bfda843daddda80d65e66d82d68c3f3fc2975b734f206313"} Oct 30 04:24:59.112103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:59.112066 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:24:59.326229 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:59.326198 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:24:59.394822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:59.394736 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-openshift-ca\") pod \"371744d4-b7e1-4021-8f03-3bc955b6aab3\" (UID: \"371744d4-b7e1-4021-8f03-3bc955b6aab3\") " Oct 30 04:24:59.395411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:59.395312 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-openshift-ca" pod "371744d4-b7e1-4021-8f03-3bc955b6aab3" (UID: "371744d4-b7e1-4021-8f03-3bc955b6aab3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:24:59.395970 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:59.395624 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371744d4-b7e1-4021-8f03-3bc955b6aab3-kube-api-access-p5trv\" (UniqueName: \"kubernetes.io/projected/371744d4-b7e1-4021-8f03-3bc955b6aab3-kube-api-access-p5trv\") pod \"371744d4-b7e1-4021-8f03-3bc955b6aab3\" (UID: \"371744d4-b7e1-4021-8f03-3bc955b6aab3\") " Oct 30 04:24:59.396350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:59.396332 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-xks-ca\") pod \"371744d4-b7e1-4021-8f03-3bc955b6aab3\" (UID: \"371744d4-b7e1-4021-8f03-3bc955b6aab3\") " Oct 30 04:24:59.397307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:59.397285 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:24:59.399840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:59.399716 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-xks-ca" pod "371744d4-b7e1-4021-8f03-3bc955b6aab3" (UID: "371744d4-b7e1-4021-8f03-3bc955b6aab3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:24:59.399840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:59.399803 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371744d4-b7e1-4021-8f03-3bc955b6aab3-kube-api-access-p5trv" pod "371744d4-b7e1-4021-8f03-3bc955b6aab3" (UID: "371744d4-b7e1-4021-8f03-3bc955b6aab3"). InnerVolumeSpecName "kube-api-access-p5trv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:24:59.499298 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:59.499263 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5trv\" (UniqueName: \"kubernetes.io/projected/371744d4-b7e1-4021-8f03-3bc955b6aab3-kube-api-access-p5trv\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:24:59.499298 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:59.499295 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371744d4-b7e1-4021-8f03-3bc955b6aab3-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:24:59.508443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:24:59.508395 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:00.158203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:00.158175 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" Oct 30 04:25:00.158410 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:00.158200 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq" event={"ID":"371744d4-b7e1-4021-8f03-3bc955b6aab3","Type":"ContainerDied","Data":"d43cd679ceaaece6bfda843daddda80d65e66d82d68c3f3fc2975b734f206313"} Oct 30 04:25:00.158410 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:00.158256 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d43cd679ceaaece6bfda843daddda80d65e66d82d68c3f3fc2975b734f206313" Oct 30 04:25:01.108836 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:01.108793 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:02.712934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:02.712899 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:04.455904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:04.455858 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:05.583476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:05.583436 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:05.754789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:05.754758 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48"] Oct 30 04:25:05.781471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:05.781435 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx"] Oct 30 04:25:05.986704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:05.986668 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:06.122678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:06.122652 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" Oct 30 04:25:06.189326 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:06.189279 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:06.416071 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:06.415688 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:07.010720 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:07.010689 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp"] Oct 30 04:25:07.341738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:07.341644 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:07.553521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:07.553451 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:08.105629 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:08.105593 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:09.846637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:09.846596 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:10.044139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:10.044101 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:10.755483 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:10.755445 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" podUID="10f1d248-2b4d-4686-a349-04bd92ee3620" containerName="payload-processing" containerID="cri-o://1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb" gracePeriod=25 Oct 30 04:25:10.781999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:10.781971 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" podUID="74c40414-822f-4a98-ba77-73c7ca563dd5" containerName="payload-pre-processing" containerID="cri-o://d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3" gracePeriod=25 Oct 30 04:25:11.101801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.101772 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" Oct 30 04:25:11.126530 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.126505 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" Oct 30 04:25:11.171392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.171359 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c40414-822f-4a98-ba77-73c7ca563dd5-kube-api-access-bbglh\" (UniqueName: \"kubernetes.io/projected/74c40414-822f-4a98-ba77-73c7ca563dd5-kube-api-access-bbglh\") pod \"74c40414-822f-4a98-ba77-73c7ca563dd5\" (UID: \"74c40414-822f-4a98-ba77-73c7ca563dd5\") " Oct 30 04:25:11.171809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.171779 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10f1d248-2b4d-4686-a349-04bd92ee3620-kube-api-access-c72rn\" (UniqueName: \"kubernetes.io/projected/10f1d248-2b4d-4686-a349-04bd92ee3620-kube-api-access-c72rn\") pod \"10f1d248-2b4d-4686-a349-04bd92ee3620\" (UID: \"10f1d248-2b4d-4686-a349-04bd92ee3620\") " Oct 30 04:25:11.171992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.171832 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10f1d248-2b4d-4686-a349-04bd92ee3620-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f1d248-2b4d-4686-a349-04bd92ee3620-plugins-config-volume\") pod \"10f1d248-2b4d-4686-a349-04bd92ee3620\" (UID: \"10f1d248-2b4d-4686-a349-04bd92ee3620\") " Oct 30 04:25:11.171992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.171904 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c40414-822f-4a98-ba77-73c7ca563dd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c40414-822f-4a98-ba77-73c7ca563dd5-plugins-config-volume\") pod \"74c40414-822f-4a98-ba77-73c7ca563dd5\" (UID: \"74c40414-822f-4a98-ba77-73c7ca563dd5\") " Oct 30 04:25:11.173102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.173011 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c40414-822f-4a98-ba77-73c7ca563dd5-plugins-config-volume" pod "74c40414-822f-4a98-ba77-73c7ca563dd5" (UID: "74c40414-822f-4a98-ba77-73c7ca563dd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:11.173102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.173056 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10f1d248-2b4d-4686-a349-04bd92ee3620-plugins-config-volume" pod "10f1d248-2b4d-4686-a349-04bd92ee3620" (UID: "10f1d248-2b4d-4686-a349-04bd92ee3620"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:11.174896 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.174869 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c40414-822f-4a98-ba77-73c7ca563dd5-kube-api-access-bbglh" pod "74c40414-822f-4a98-ba77-73c7ca563dd5" (UID: "74c40414-822f-4a98-ba77-73c7ca563dd5"). InnerVolumeSpecName "kube-api-access-bbglh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:11.175595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.175573 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10f1d248-2b4d-4686-a349-04bd92ee3620-kube-api-access-c72rn" pod "10f1d248-2b4d-4686-a349-04bd92ee3620" (UID: "10f1d248-2b4d-4686-a349-04bd92ee3620"). InnerVolumeSpecName "kube-api-access-c72rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:11.260446 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:25:11.254019 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:25:11.280502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.272055 4793 generic.go:356] "Generic (PLEG): container finished" podID="74c40414-822f-4a98-ba77-73c7ca563dd5" containerID="d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3" exitCode=0 Oct 30 04:25:11.280502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.272174 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" event={"ID":"74c40414-822f-4a98-ba77-73c7ca563dd5","Type":"ContainerDied","Data":"d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3"} Oct 30 04:25:11.280502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.272219 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" event={"ID":"74c40414-822f-4a98-ba77-73c7ca563dd5","Type":"ContainerDied","Data":"6c1abdc78e92589cdfe1f7df31e620b1a64a9d93dc39cc27c2b42dcd0c9c6665"} Oct 30 04:25:11.280502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.272242 4793 scope.go:122] "RemoveContainer" containerID="d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3" Oct 30 04:25:11.280502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.272396 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx" Oct 30 04:25:11.282598 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.282571 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbglh\" (UniqueName: \"kubernetes.io/projected/74c40414-822f-4a98-ba77-73c7ca563dd5-kube-api-access-bbglh\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:11.282828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.282812 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c72rn\" (UniqueName: \"kubernetes.io/projected/10f1d248-2b4d-4686-a349-04bd92ee3620-kube-api-access-c72rn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:11.282950 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.282938 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10f1d248-2b4d-4686-a349-04bd92ee3620-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:11.283095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.283081 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c40414-822f-4a98-ba77-73c7ca563dd5-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:11.289141 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.288537 4793 generic.go:356] "Generic (PLEG): container finished" podID="10f1d248-2b4d-4686-a349-04bd92ee3620" containerID="1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb" exitCode=0 Oct 30 04:25:11.289141 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.289087 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" event={"ID":"10f1d248-2b4d-4686-a349-04bd92ee3620","Type":"ContainerDied","Data":"1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb"} Oct 30 04:25:11.289464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.289361 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" event={"ID":"10f1d248-2b4d-4686-a349-04bd92ee3620","Type":"ContainerDied","Data":"b7786bf16a0f3a24955b9f65a6ba1938babf2e4b09a7775d9a4bceaf600c60a9"} Oct 30 04:25:11.290472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.289609 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48" Oct 30 04:25:11.304613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.304597 4793 scope.go:122] "RemoveContainer" containerID="d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3" Oct 30 04:25:11.305319 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:25:11.305295 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3\": container with ID starting with d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3 not found: ID does not exist" containerID="d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3" Oct 30 04:25:11.305769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.305740 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3"} err="failed to get container status \"d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3\": rpc error: code = NotFound desc = could not find container \"d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3\": container with ID starting with d726ddf33a35f6112d5ca2f435d2435beaedf9ec7e3c1a04a473fa2ca47e6af3 not found: ID does not exist" Oct 30 04:25:11.305769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.305769 4793 scope.go:122] "RemoveContainer" containerID="1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb" Oct 30 04:25:11.315594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.315570 4793 scope.go:122] "RemoveContainer" containerID="1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb" Oct 30 04:25:11.315847 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:25:11.315826 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb\": container with ID starting with 1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb not found: ID does not exist" containerID="1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb" Oct 30 04:25:11.315905 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.315854 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb"} err="failed to get container status \"1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb\": rpc error: code = NotFound desc = could not find container \"1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb\": container with ID starting with 1038af3c91d6a59948b4f90d75540629a4352250a946959bc8ff5e582ac41ecb not found: ID does not exist" Oct 30 04:25:11.338524 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.338501 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx"] Oct 30 04:25:11.343155 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.343137 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e124dc-96dfb687-6k6rx"] Oct 30 04:25:11.394628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.394599 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48"] Oct 30 04:25:11.398040 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.398013 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e124dc-7c47449b48-f5r48"] Oct 30 04:25:11.603321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.603229 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:11.804326 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.804287 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:11.967095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.967067 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph"] Oct 30 04:25:11.967764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.967744 4793 cpu_manager.go:414] "containerMap: removing container" podUID="74c40414-822f-4a98-ba77-73c7ca563dd5" containerName="payload-pre-processing" Oct 30 04:25:11.967764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.967764 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c40414-822f-4a98-ba77-73c7ca563dd5" containerName="payload-pre-processing" Oct 30 04:25:11.967863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.967775 4793 cpu_manager.go:414] "containerMap: removing container" podUID="10f1d248-2b4d-4686-a349-04bd92ee3620" containerName="payload-processing" Oct 30 04:25:11.967863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.967780 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10f1d248-2b4d-4686-a349-04bd92ee3620" containerName="payload-processing" Oct 30 04:25:11.967863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.967796 4793 cpu_manager.go:414] "containerMap: removing container" podUID="371744d4-b7e1-4021-8f03-3bc955b6aab3" containerName="revoke-keys" Oct 30 04:25:11.967863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.967801 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371744d4-b7e1-4021-8f03-3bc955b6aab3" containerName="revoke-keys" Oct 30 04:25:11.968034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.967912 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c40414-822f-4a98-ba77-73c7ca563dd5" containerName="payload-pre-processing" Oct 30 04:25:11.968034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.967919 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="10f1d248-2b4d-4686-a349-04bd92ee3620" containerName="payload-processing" Oct 30 04:25:11.968034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.967926 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="371744d4-b7e1-4021-8f03-3bc955b6aab3" containerName="revoke-keys" Oct 30 04:25:11.972357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.972340 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:11.987831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.987807 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:25:11.987951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.987919 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:25:11.988045 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.987983 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:25:11.988045 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.988019 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:25:11.988150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.988046 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph"] Oct 30 04:25:11.991860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.991836 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe861886-8ffph\" (UID: \"a0189496-3809-4080-b12b-42152c45e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:11.991993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.991968 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppbx\" (UniqueName: \"kubernetes.io/projected/a0189496-3809-4080-b12b-42152c45e7b1-kube-api-access-sppbx\") pod \"maas-api-revoke-sub-fe861886-8ffph\" (UID: \"a0189496-3809-4080-b12b-42152c45e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:11.992172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:11.992154 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe861886-8ffph\" (UID: \"a0189496-3809-4080-b12b-42152c45e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:12.011643 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:12.011608 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" podUID="d9857c7f-c96b-4057-8c34-7359cf82ffef" containerName="payload-processing" containerID="cri-o://03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea" gracePeriod=25 Oct 30 04:25:12.093852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:12.093823 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe861886-8ffph\" (UID: \"a0189496-3809-4080-b12b-42152c45e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:12.093969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:12.093886 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppbx\" (UniqueName: \"kubernetes.io/projected/a0189496-3809-4080-b12b-42152c45e7b1-kube-api-access-sppbx\") pod \"maas-api-revoke-sub-fe861886-8ffph\" (UID: \"a0189496-3809-4080-b12b-42152c45e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:12.094045 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:12.094029 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe861886-8ffph\" (UID: \"a0189496-3809-4080-b12b-42152c45e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:12.094646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:12.094626 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe861886-8ffph\" (UID: \"a0189496-3809-4080-b12b-42152c45e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:12.096166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:12.096144 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe861886-8ffph\" (UID: \"a0189496-3809-4080-b12b-42152c45e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:12.117556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:12.117534 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppbx\" (UniqueName: \"kubernetes.io/projected/a0189496-3809-4080-b12b-42152c45e7b1-kube-api-access-sppbx\") pod \"maas-api-revoke-sub-fe861886-8ffph\" (UID: \"a0189496-3809-4080-b12b-42152c45e7b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:12.285830 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:12.285760 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:12.592556 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:25:12.592510 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0189496_3809_4080_b12b_42152c45e7b1.slice/crio-ff86fd722fb00830d9e10c04af6884ea738a413ae4ecd42668daf3c3ac00576d WatchSource:0}: Error finding container ff86fd722fb00830d9e10c04af6884ea738a413ae4ecd42668daf3c3ac00576d: Status 404 returned error can't find the container with id ff86fd722fb00830d9e10c04af6884ea738a413ae4ecd42668daf3c3ac00576d Oct 30 04:25:12.593228 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:12.593202 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph"] Oct 30 04:25:12.860527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:12.860491 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10f1d248-2b4d-4686-a349-04bd92ee3620" path="/var/lib/kubelet/pods/10f1d248-2b4d-4686-a349-04bd92ee3620/volumes" Oct 30 04:25:12.861131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:12.861110 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c40414-822f-4a98-ba77-73c7ca563dd5" path="/var/lib/kubelet/pods/74c40414-822f-4a98-ba77-73c7ca563dd5/volumes" Oct 30 04:25:13.307838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:13.307802 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" event={"ID":"a0189496-3809-4080-b12b-42152c45e7b1","Type":"ContainerStarted","Data":"7ff5ad862c0ae0d369c1e9d138b0eeffb44cd579a5aebac7728ac99911f8064f"} Oct 30 04:25:13.308319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:13.307857 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" event={"ID":"a0189496-3809-4080-b12b-42152c45e7b1","Type":"ContainerStarted","Data":"ff86fd722fb00830d9e10c04af6884ea738a413ae4ecd42668daf3c3ac00576d"} Oct 30 04:25:13.351769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:13.351701 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" podStartSLOduration=2.351681957 podStartE2EDuration="2.351681957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:25:13.348873662 +0000 UTC m=+1636.634748458" watchObservedRunningTime="2026-10-30 04:25:13.351681957 +0000 UTC m=+1636.637556734" Oct 30 04:25:13.403310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:13.402038 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:14.317312 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:14.317276 4793 generic.go:356] "Generic (PLEG): container finished" podID="a0189496-3809-4080-b12b-42152c45e7b1" containerID="7ff5ad862c0ae0d369c1e9d138b0eeffb44cd579a5aebac7728ac99911f8064f" exitCode=0 Oct 30 04:25:14.317819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:14.317323 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" event={"ID":"a0189496-3809-4080-b12b-42152c45e7b1","Type":"ContainerDied","Data":"7ff5ad862c0ae0d369c1e9d138b0eeffb44cd579a5aebac7728ac99911f8064f"} Oct 30 04:25:15.034145 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:15.034093 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:15.482127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:15.482101 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:15.541102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:15.541062 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0189496-3809-4080-b12b-42152c45e7b1-kube-api-access-sppbx\" (UniqueName: \"kubernetes.io/projected/a0189496-3809-4080-b12b-42152c45e7b1-kube-api-access-sppbx\") pod \"a0189496-3809-4080-b12b-42152c45e7b1\" (UID: \"a0189496-3809-4080-b12b-42152c45e7b1\") " Oct 30 04:25:15.541347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:15.541325 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-xks-ca\") pod \"a0189496-3809-4080-b12b-42152c45e7b1\" (UID: \"a0189496-3809-4080-b12b-42152c45e7b1\") " Oct 30 04:25:15.541403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:15.541361 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-openshift-ca\") pod \"a0189496-3809-4080-b12b-42152c45e7b1\" (UID: \"a0189496-3809-4080-b12b-42152c45e7b1\") " Oct 30 04:25:15.541824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:15.541794 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-openshift-ca" pod "a0189496-3809-4080-b12b-42152c45e7b1" (UID: "a0189496-3809-4080-b12b-42152c45e7b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:15.542047 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:15.542026 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:15.543348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:15.543314 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0189496-3809-4080-b12b-42152c45e7b1-kube-api-access-sppbx" pod "a0189496-3809-4080-b12b-42152c45e7b1" (UID: "a0189496-3809-4080-b12b-42152c45e7b1"). InnerVolumeSpecName "kube-api-access-sppbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:15.543483 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:15.543438 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-xks-ca" pod "a0189496-3809-4080-b12b-42152c45e7b1" (UID: "a0189496-3809-4080-b12b-42152c45e7b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:25:15.643191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:15.643127 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0189496-3809-4080-b12b-42152c45e7b1-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:15.643191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:15.643153 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sppbx\" (UniqueName: \"kubernetes.io/projected/a0189496-3809-4080-b12b-42152c45e7b1-kube-api-access-sppbx\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:15.985279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:15.985245 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" podUID="d9857c7f-c96b-4057-8c34-7359cf82ffef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:25:16.334388 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:16.334318 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" Oct 30 04:25:16.334388 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:16.334352 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph" event={"ID":"a0189496-3809-4080-b12b-42152c45e7b1","Type":"ContainerDied","Data":"ff86fd722fb00830d9e10c04af6884ea738a413ae4ecd42668daf3c3ac00576d"} Oct 30 04:25:16.334585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:16.334401 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff86fd722fb00830d9e10c04af6884ea738a413ae4ecd42668daf3c3ac00576d" Oct 30 04:25:16.692628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:16.692591 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:18.486654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:18.486617 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:20.322447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:20.322386 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:21.793675 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:21.793626 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:22.253443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.253257 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th"] Oct 30 04:25:22.285285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.285249 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a0189496-3809-4080-b12b-42152c45e7b1" containerName="revoke-keys" Oct 30 04:25:22.285537 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.285517 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0189496-3809-4080-b12b-42152c45e7b1" containerName="revoke-keys" Oct 30 04:25:22.285846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.285827 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0189496-3809-4080-b12b-42152c45e7b1" containerName="revoke-keys" Oct 30 04:25:22.305857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.302455 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th"] Oct 30 04:25:22.305857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.302628 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:22.319169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.319150 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:25:22.331398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.331382 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:25:22.331496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.331410 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:25:22.331594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.331580 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:25:22.352067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.352044 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9736ddd-hx5th\" (UID: \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:22.352209 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.352192 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9736ddd-hx5th\" (UID: \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:22.352252 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.352240 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74ngb\" (UniqueName: \"kubernetes.io/projected/36778c6c-6fb6-4ce9-9ca5-0bda78427209-kube-api-access-74ngb\") pod \"maas-api-revoke-sub-f9736ddd-hx5th\" (UID: \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:22.401387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.401352 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:22.453049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.453021 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9736ddd-hx5th\" (UID: \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:22.453190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.453166 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9736ddd-hx5th\" (UID: \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:22.453337 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.453249 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74ngb\" (UniqueName: \"kubernetes.io/projected/36778c6c-6fb6-4ce9-9ca5-0bda78427209-kube-api-access-74ngb\") pod \"maas-api-revoke-sub-f9736ddd-hx5th\" (UID: \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:22.461445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.453880 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9736ddd-hx5th\" (UID: \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:22.518026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.509914 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9736ddd-hx5th\" (UID: \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:22.518026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.510515 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74ngb\" (UniqueName: \"kubernetes.io/projected/36778c6c-6fb6-4ce9-9ca5-0bda78427209-kube-api-access-74ngb\") pod \"maas-api-revoke-sub-f9736ddd-hx5th\" (UID: \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:22.615338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.615288 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:22.619215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.619184 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:22.979454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:22.978780 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th"] Oct 30 04:25:23.418651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:23.418624 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" event={"ID":"36778c6c-6fb6-4ce9-9ca5-0bda78427209","Type":"ContainerStarted","Data":"0f08333d21f8792cc1589241a4a7ff207e1e2a25574c4e49850606f4c3dea852"} Oct 30 04:25:23.418790 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:23.418680 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" event={"ID":"36778c6c-6fb6-4ce9-9ca5-0bda78427209","Type":"ContainerStarted","Data":"e429d122bf14da80105b8a0b62d0dc47b91ee01b1b69ba9d0bed52f2eb54d401"} Oct 30 04:25:23.494930 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:23.494867 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" podStartSLOduration=1.494848126 podStartE2EDuration="1.494848126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:25:23.471648832 +0000 UTC m=+1646.757523609" watchObservedRunningTime="2026-10-30 04:25:23.494848126 +0000 UTC m=+1646.780722902" Oct 30 04:25:24.191008 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:24.190975 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:24.437224 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:24.437143 4793 generic.go:356] "Generic (PLEG): container finished" podID="36778c6c-6fb6-4ce9-9ca5-0bda78427209" containerID="0f08333d21f8792cc1589241a4a7ff207e1e2a25574c4e49850606f4c3dea852" exitCode=0 Oct 30 04:25:24.437352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:24.437230 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" event={"ID":"36778c6c-6fb6-4ce9-9ca5-0bda78427209","Type":"ContainerDied","Data":"0f08333d21f8792cc1589241a4a7ff207e1e2a25574c4e49850606f4c3dea852"} Oct 30 04:25:25.623878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:25.623846 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:25.727751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:25.727640 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-xks-ca\") pod \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\" (UID: \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\") " Oct 30 04:25:25.727751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:25.727747 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36778c6c-6fb6-4ce9-9ca5-0bda78427209-kube-api-access-74ngb\" (UniqueName: \"kubernetes.io/projected/36778c6c-6fb6-4ce9-9ca5-0bda78427209-kube-api-access-74ngb\") pod \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\" (UID: \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\") " Oct 30 04:25:25.728031 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:25.727982 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-openshift-ca\") pod \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\" (UID: \"36778c6c-6fb6-4ce9-9ca5-0bda78427209\") " Oct 30 04:25:25.728682 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:25.728627 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-openshift-ca" pod "36778c6c-6fb6-4ce9-9ca5-0bda78427209" (UID: "36778c6c-6fb6-4ce9-9ca5-0bda78427209"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:25.730347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:25.730305 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-xks-ca" pod "36778c6c-6fb6-4ce9-9ca5-0bda78427209" (UID: "36778c6c-6fb6-4ce9-9ca5-0bda78427209"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:25:25.730522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:25.730469 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36778c6c-6fb6-4ce9-9ca5-0bda78427209-kube-api-access-74ngb" pod "36778c6c-6fb6-4ce9-9ca5-0bda78427209" (UID: "36778c6c-6fb6-4ce9-9ca5-0bda78427209"). InnerVolumeSpecName "kube-api-access-74ngb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:25.830156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:25.830120 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:25.830156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:25.830156 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36778c6c-6fb6-4ce9-9ca5-0bda78427209-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:25.830390 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:25.830171 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74ngb\" (UniqueName: \"kubernetes.io/projected/36778c6c-6fb6-4ce9-9ca5-0bda78427209-kube-api-access-74ngb\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:25.922055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:25.922015 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:25.984691 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:25.984594 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" podUID="d9857c7f-c96b-4057-8c34-7359cf82ffef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:25:26.455719 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:26.455689 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" Oct 30 04:25:26.455879 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:26.455686 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th" event={"ID":"36778c6c-6fb6-4ce9-9ca5-0bda78427209","Type":"ContainerDied","Data":"e429d122bf14da80105b8a0b62d0dc47b91ee01b1b69ba9d0bed52f2eb54d401"} Oct 30 04:25:26.455879 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:26.455829 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e429d122bf14da80105b8a0b62d0dc47b91ee01b1b69ba9d0bed52f2eb54d401" Oct 30 04:25:27.269099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:27.269060 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:27.481368 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:27.481330 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:27.706980 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:27.706941 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:27.972846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:27.972716 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:28.378540 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.375440 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:28.484460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.484408 4793 generic.go:356] "Generic (PLEG): container finished" podID="e5e56157-cea5-4efb-86a3-797f64913b05" containerID="3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7" exitCode=0 Oct 30 04:25:28.484637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.484483 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" event={"ID":"e5e56157-cea5-4efb-86a3-797f64913b05","Type":"ContainerDied","Data":"3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7"} Oct 30 04:25:28.485001 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.484983 4793 scope.go:122] "RemoveContainer" containerID="3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7" Oct 30 04:25:28.486640 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.486613 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:25:28.486708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.486685 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:25:28.486748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.486718 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:25:28.672176 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.672104 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr"] Oct 30 04:25:28.672789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.672773 4793 cpu_manager.go:414] "containerMap: removing container" podUID="36778c6c-6fb6-4ce9-9ca5-0bda78427209" containerName="revoke-keys" Oct 30 04:25:28.672854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.672790 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36778c6c-6fb6-4ce9-9ca5-0bda78427209" containerName="revoke-keys" Oct 30 04:25:28.672907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.672898 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="36778c6c-6fb6-4ce9-9ca5-0bda78427209" containerName="revoke-keys" Oct 30 04:25:28.675384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.675365 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:28.691239 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.691211 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:25:28.693801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.693579 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:25:28.693801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.693735 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:25:28.693977 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.693857 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:25:28.695454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.695404 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr"] Oct 30 04:25:28.796130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.796093 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c899505-trvbr\" (UID: \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:28.796325 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.796281 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c899505-trvbr\" (UID: \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:28.796325 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.796316 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cz65\" (UniqueName: \"kubernetes.io/projected/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-kube-api-access-6cz65\") pod \"maas-api-revoke-sub-4c899505-trvbr\" (UID: \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:28.897761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.897724 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c899505-trvbr\" (UID: \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:28.897955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.897883 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c899505-trvbr\" (UID: \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:28.897955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.897904 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cz65\" (UniqueName: \"kubernetes.io/projected/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-kube-api-access-6cz65\") pod \"maas-api-revoke-sub-4c899505-trvbr\" (UID: \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:28.899045 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.899017 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c899505-trvbr\" (UID: \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:28.900669 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.900645 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c899505-trvbr\" (UID: \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:28.921959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.921921 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cz65\" (UniqueName: \"kubernetes.io/projected/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-kube-api-access-6cz65\") pod \"maas-api-revoke-sub-4c899505-trvbr\" (UID: \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:28.980349 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.980313 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:28.988194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:28.988162 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:29.186744 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:29.186696 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:29.336502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:29.336472 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr"] Oct 30 04:25:29.337016 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:25:29.336991 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379aab4d_7a33_4ca0_bbd7_8ffb091638d5.slice/crio-3a9a96cd55d599e0c859f43e8c518730a555ce74c8df7bdb94bcae40023716ee WatchSource:0}: Error finding container 3a9a96cd55d599e0c859f43e8c518730a555ce74c8df7bdb94bcae40023716ee: Status 404 returned error can't find the container with id 3a9a96cd55d599e0c859f43e8c518730a555ce74c8df7bdb94bcae40023716ee Oct 30 04:25:29.499373 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:29.499340 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" event={"ID":"e5e56157-cea5-4efb-86a3-797f64913b05","Type":"ContainerStarted","Data":"073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6"} Oct 30 04:25:29.501744 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:29.501711 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" event={"ID":"379aab4d-7a33-4ca0-bbd7-8ffb091638d5","Type":"ContainerStarted","Data":"3a9a96cd55d599e0c859f43e8c518730a555ce74c8df7bdb94bcae40023716ee"} Oct 30 04:25:29.540049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:29.540010 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:30.088997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:30.088922 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:25:30.089151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:30.089128 4793 patch_prober.go:29] interesting pod/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 04:25:30.089196 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:30.089179 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 04:25:30.510353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:30.510323 4793 generic.go:356] "Generic (PLEG): container finished" podID="379aab4d-7a33-4ca0-bbd7-8ffb091638d5" containerID="06db8207b43b40f455bbb94836bc98806be8371053e86b10928f63502124835e" exitCode=0 Oct 30 04:25:30.510677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:30.510406 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" event={"ID":"379aab4d-7a33-4ca0-bbd7-8ffb091638d5","Type":"ContainerDied","Data":"06db8207b43b40f455bbb94836bc98806be8371053e86b10928f63502124835e"} Oct 30 04:25:31.088907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.088873 4793 patch_prober.go:29] interesting pod/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 04:25:31.089083 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.088927 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 04:25:31.318782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.318541 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:31.716169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.716139 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:31.739749 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.739722 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-openshift-ca\") pod \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\" (UID: \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\") " Oct 30 04:25:31.739968 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.739949 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-xks-ca\") pod \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\" (UID: \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\") " Oct 30 04:25:31.740029 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.740005 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-openshift-ca" pod "379aab4d-7a33-4ca0-bbd7-8ffb091638d5" (UID: "379aab4d-7a33-4ca0-bbd7-8ffb091638d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:31.740087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.740025 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-kube-api-access-6cz65\" (UniqueName: \"kubernetes.io/projected/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-kube-api-access-6cz65\") pod \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\" (UID: \"379aab4d-7a33-4ca0-bbd7-8ffb091638d5\") " Oct 30 04:25:31.740566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.740545 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:31.742078 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.742056 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-xks-ca" pod "379aab4d-7a33-4ca0-bbd7-8ffb091638d5" (UID: "379aab4d-7a33-4ca0-bbd7-8ffb091638d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:25:31.742245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.742225 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-kube-api-access-6cz65" pod "379aab4d-7a33-4ca0-bbd7-8ffb091638d5" (UID: "379aab4d-7a33-4ca0-bbd7-8ffb091638d5"). InnerVolumeSpecName "kube-api-access-6cz65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:31.848176 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.848087 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:31.848401 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:31.848376 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cz65\" (UniqueName: \"kubernetes.io/projected/379aab4d-7a33-4ca0-bbd7-8ffb091638d5-kube-api-access-6cz65\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:32.089051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.089017 4793 patch_prober.go:29] interesting pod/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 04:25:32.089294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.089259 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 04:25:32.280438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.277103 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:32.302358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.302328 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" Oct 30 04:25:32.382269 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.382187 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9857c7f-c96b-4057-8c34-7359cf82ffef-kube-api-access-mzg98\" (UniqueName: \"kubernetes.io/projected/d9857c7f-c96b-4057-8c34-7359cf82ffef-kube-api-access-mzg98\") pod \"d9857c7f-c96b-4057-8c34-7359cf82ffef\" (UID: \"d9857c7f-c96b-4057-8c34-7359cf82ffef\") " Oct 30 04:25:32.382490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.382467 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9857c7f-c96b-4057-8c34-7359cf82ffef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9857c7f-c96b-4057-8c34-7359cf82ffef-plugins-config-volume\") pod \"d9857c7f-c96b-4057-8c34-7359cf82ffef\" (UID: \"d9857c7f-c96b-4057-8c34-7359cf82ffef\") " Oct 30 04:25:32.383907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.383870 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9857c7f-c96b-4057-8c34-7359cf82ffef-plugins-config-volume" pod "d9857c7f-c96b-4057-8c34-7359cf82ffef" (UID: "d9857c7f-c96b-4057-8c34-7359cf82ffef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:32.384981 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.384952 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9857c7f-c96b-4057-8c34-7359cf82ffef-kube-api-access-mzg98" pod "d9857c7f-c96b-4057-8c34-7359cf82ffef" (UID: "d9857c7f-c96b-4057-8c34-7359cf82ffef"). InnerVolumeSpecName "kube-api-access-mzg98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:32.484774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.484739 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9857c7f-c96b-4057-8c34-7359cf82ffef-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:32.484774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.484770 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzg98\" (UniqueName: \"kubernetes.io/projected/d9857c7f-c96b-4057-8c34-7359cf82ffef-kube-api-access-mzg98\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:32.528173 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.528139 4793 generic.go:356] "Generic (PLEG): container finished" podID="d9857c7f-c96b-4057-8c34-7359cf82ffef" containerID="03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea" exitCode=0 Oct 30 04:25:32.528340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.528189 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" Oct 30 04:25:32.528340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.528225 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" event={"ID":"d9857c7f-c96b-4057-8c34-7359cf82ffef","Type":"ContainerDied","Data":"03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea"} Oct 30 04:25:32.528340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.528278 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp" event={"ID":"d9857c7f-c96b-4057-8c34-7359cf82ffef","Type":"ContainerDied","Data":"1e3c4c7d20e0556ba631b6c94397701fb61472c635a057ed33eb46fca568a26f"} Oct 30 04:25:32.528340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.528305 4793 scope.go:122] "RemoveContainer" containerID="03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea" Oct 30 04:25:32.530025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.530000 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" event={"ID":"379aab4d-7a33-4ca0-bbd7-8ffb091638d5","Type":"ContainerDied","Data":"3a9a96cd55d599e0c859f43e8c518730a555ce74c8df7bdb94bcae40023716ee"} Oct 30 04:25:32.530025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.530021 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr" Oct 30 04:25:32.530233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.530045 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a9a96cd55d599e0c859f43e8c518730a555ce74c8df7bdb94bcae40023716ee" Oct 30 04:25:32.539904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.539868 4793 scope.go:122] "RemoveContainer" containerID="03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea" Oct 30 04:25:32.540307 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:25:32.540167 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea\": container with ID starting with 03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea not found: ID does not exist" containerID="03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea" Oct 30 04:25:32.540307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.540198 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea"} err="failed to get container status \"03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea\": rpc error: code = NotFound desc = could not find container \"03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea\": container with ID starting with 03ca763856ef948089ce8aac7ba3a6c707b53c9f1090fabbd22cc318bdf545ea not found: ID does not exist" Oct 30 04:25:32.588733 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.588701 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp"] Oct 30 04:25:32.592843 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.592811 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1a0d3427-7c69dfbb76-474hp"] Oct 30 04:25:32.847485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:32.847457 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9857c7f-c96b-4057-8c34-7359cf82ffef" path="/var/lib/kubelet/pods/d9857c7f-c96b-4057-8c34-7359cf82ffef/volumes" Oct 30 04:25:33.011548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:33.010481 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:33.089346 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:33.089313 4793 patch_prober.go:29] interesting pod/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 04:25:33.089550 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:33.089378 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 04:25:33.939574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:33.939536 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:34.088949 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:34.088916 4793 patch_prober.go:29] interesting pod/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Oct 30 04:25:34.089120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:34.088982 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Oct 30 04:25:34.704160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:34.704045 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:35.032884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:35.032810 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:25:35.093881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:35.093853 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:25:35.387913 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:35.387836 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj"] Oct 30 04:25:35.420219 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:35.420191 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz"] Oct 30 04:25:35.554011 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:35.553982 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:25:35.668384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:35.656105 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:35.860453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:35.859496 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:36.297659 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:36.297626 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:36.633832 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:36.633740 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:38.307899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:38.307861 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:40.126033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:40.125945 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:40.392501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:40.372723 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4"] Oct 30 04:25:40.430448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:40.420531 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" podUID="1d74fd83-dd2d-4b3e-8269-b96ff52ed274" containerName="payload-pre-processing" containerID="cri-o://90d70676903950852c71bebf9115a25bd5f8ed3f5380a797e6734a2ac04ddbb1" gracePeriod=25 Oct 30 04:25:40.430448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:40.429119 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" podUID="025f0ebe-c1e2-439f-8072-8543c281269a" containerName="payload-processing" containerID="cri-o://9bd8174456efa9ac08c81d4dd77aa0565e4146a168c1a4d60079c4a0c4d42221" gracePeriod=25 Oct 30 04:25:40.568494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:40.568399 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:40.597815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:40.597412 4793 generic.go:356] "Generic (PLEG): container finished" podID="025f0ebe-c1e2-439f-8072-8543c281269a" containerID="9bd8174456efa9ac08c81d4dd77aa0565e4146a168c1a4d60079c4a0c4d42221" exitCode=0 Oct 30 04:25:40.597815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:40.597730 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" event={"ID":"025f0ebe-c1e2-439f-8072-8543c281269a","Type":"ContainerDied","Data":"9bd8174456efa9ac08c81d4dd77aa0565e4146a168c1a4d60079c4a0c4d42221"} Oct 30 04:25:40.600257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:40.600115 4793 generic.go:356] "Generic (PLEG): container finished" podID="1d74fd83-dd2d-4b3e-8269-b96ff52ed274" containerID="90d70676903950852c71bebf9115a25bd5f8ed3f5380a797e6734a2ac04ddbb1" exitCode=0 Oct 30 04:25:40.600257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:40.600190 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" event={"ID":"1d74fd83-dd2d-4b3e-8269-b96ff52ed274","Type":"ContainerDied","Data":"90d70676903950852c71bebf9115a25bd5f8ed3f5380a797e6734a2ac04ddbb1"} Oct 30 04:25:40.790663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:40.790329 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:40.885892 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:40.885596 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" Oct 30 04:25:40.889875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:40.889774 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" Oct 30 04:25:41.069443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.064087 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-plugins-config-volume\") pod \"1d74fd83-dd2d-4b3e-8269-b96ff52ed274\" (UID: \"1d74fd83-dd2d-4b3e-8269-b96ff52ed274\") " Oct 30 04:25:41.069443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.064151 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/025f0ebe-c1e2-439f-8072-8543c281269a-kube-api-access-98d8m\" (UniqueName: \"kubernetes.io/projected/025f0ebe-c1e2-439f-8072-8543c281269a-kube-api-access-98d8m\") pod \"025f0ebe-c1e2-439f-8072-8543c281269a\" (UID: \"025f0ebe-c1e2-439f-8072-8543c281269a\") " Oct 30 04:25:41.069443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.064265 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-kube-api-access-5cblf\" (UniqueName: \"kubernetes.io/projected/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-kube-api-access-5cblf\") pod \"1d74fd83-dd2d-4b3e-8269-b96ff52ed274\" (UID: \"1d74fd83-dd2d-4b3e-8269-b96ff52ed274\") " Oct 30 04:25:41.069443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.064320 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/025f0ebe-c1e2-439f-8072-8543c281269a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025f0ebe-c1e2-439f-8072-8543c281269a-plugins-config-volume\") pod \"025f0ebe-c1e2-439f-8072-8543c281269a\" (UID: \"025f0ebe-c1e2-439f-8072-8543c281269a\") " Oct 30 04:25:41.069443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.065965 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-plugins-config-volume" pod "1d74fd83-dd2d-4b3e-8269-b96ff52ed274" (UID: "1d74fd83-dd2d-4b3e-8269-b96ff52ed274"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:41.069443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.066383 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/025f0ebe-c1e2-439f-8072-8543c281269a-plugins-config-volume" pod "025f0ebe-c1e2-439f-8072-8543c281269a" (UID: "025f0ebe-c1e2-439f-8072-8543c281269a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:41.108546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.107714 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/025f0ebe-c1e2-439f-8072-8543c281269a-kube-api-access-98d8m" pod "025f0ebe-c1e2-439f-8072-8543c281269a" (UID: "025f0ebe-c1e2-439f-8072-8543c281269a"). InnerVolumeSpecName "kube-api-access-98d8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:41.122073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.122011 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-kube-api-access-5cblf" pod "1d74fd83-dd2d-4b3e-8269-b96ff52ed274" (UID: "1d74fd83-dd2d-4b3e-8269-b96ff52ed274"). InnerVolumeSpecName "kube-api-access-5cblf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:41.166144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.166110 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98d8m\" (UniqueName: \"kubernetes.io/projected/025f0ebe-c1e2-439f-8072-8543c281269a-kube-api-access-98d8m\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:41.166144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.166137 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cblf\" (UniqueName: \"kubernetes.io/projected/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-kube-api-access-5cblf\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:41.166144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.166149 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/025f0ebe-c1e2-439f-8072-8543c281269a-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:41.166779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.166157 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d74fd83-dd2d-4b3e-8269-b96ff52ed274-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:41.225928 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:25:41.225889 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:25:41.265993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.265955 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:41.641930 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.641903 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" Oct 30 04:25:41.642367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.642346 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz" event={"ID":"1d74fd83-dd2d-4b3e-8269-b96ff52ed274","Type":"ContainerDied","Data":"bbd45ea8f73b29771a8db119d2e9b75e61942cb3255ecf3dff495e5b8d72b996"} Oct 30 04:25:41.642413 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.642401 4793 scope.go:122] "RemoveContainer" containerID="90d70676903950852c71bebf9115a25bd5f8ed3f5380a797e6734a2ac04ddbb1" Oct 30 04:25:41.644301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.644279 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" Oct 30 04:25:41.644443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.644303 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj" event={"ID":"025f0ebe-c1e2-439f-8072-8543c281269a","Type":"ContainerDied","Data":"0674fc348e3f6a35c81dd41cc81ed878227608ca31a5b57992015e33dd7d9f58"} Oct 30 04:25:41.653348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.653331 4793 scope.go:122] "RemoveContainer" containerID="9bd8174456efa9ac08c81d4dd77aa0565e4146a168c1a4d60079c4a0c4d42221" Oct 30 04:25:41.734451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.733743 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz"] Oct 30 04:25:41.741275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.741255 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0c530e-98c979bbc-hk7vz"] Oct 30 04:25:41.800486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.799474 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj"] Oct 30 04:25:41.828438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:41.810459 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0c530e-64d8846c56-w64nj"] Oct 30 04:25:42.508441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:42.500400 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:42.717273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:42.717234 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:42.847039 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:42.846959 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="025f0ebe-c1e2-439f-8072-8543c281269a" path="/var/lib/kubelet/pods/025f0ebe-c1e2-439f-8072-8543c281269a/volumes" Oct 30 04:25:42.847393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:42.847378 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d74fd83-dd2d-4b3e-8269-b96ff52ed274" path="/var/lib/kubelet/pods/1d74fd83-dd2d-4b3e-8269-b96ff52ed274/volumes" Oct 30 04:25:44.072096 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:44.072064 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:44.471313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:44.471275 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:44.648482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:44.648451 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq"] Oct 30 04:25:44.660835 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:44.660808 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1a0d3427-hxwhq"] Oct 30 04:25:44.679783 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:44.679744 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:44.904533 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:44.883171 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371744d4-b7e1-4021-8f03-3bc955b6aab3" path="/var/lib/kubelet/pods/371744d4-b7e1-4021-8f03-3bc955b6aab3/volumes" Oct 30 04:25:44.904533 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:44.890308 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:45.262143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.262107 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:45.373280 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.373232 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" podUID="31400a77-11ef-4706-999a-5f60cee5ecf7" containerName="payload-pre-processing" containerID="cri-o://9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be" gracePeriod=25 Oct 30 04:25:45.643988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.643963 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" Oct 30 04:25:45.740982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.740950 4793 generic.go:356] "Generic (PLEG): container finished" podID="31400a77-11ef-4706-999a-5f60cee5ecf7" containerID="9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be" exitCode=0 Oct 30 04:25:45.741157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.740997 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" Oct 30 04:25:45.741157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.741028 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" event={"ID":"31400a77-11ef-4706-999a-5f60cee5ecf7","Type":"ContainerDied","Data":"9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be"} Oct 30 04:25:45.741157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.741084 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4" event={"ID":"31400a77-11ef-4706-999a-5f60cee5ecf7","Type":"ContainerDied","Data":"ced072ad42d20d0bce3dfbafc82e0eec30ed368924a45773138aa1ff85abf627"} Oct 30 04:25:45.741157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.741107 4793 scope.go:122] "RemoveContainer" containerID="9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be" Oct 30 04:25:45.751725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.751706 4793 scope.go:122] "RemoveContainer" containerID="9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be" Oct 30 04:25:45.751993 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:25:45.751975 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be\": container with ID starting with 9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be not found: ID does not exist" containerID="9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be" Oct 30 04:25:45.752038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.752001 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be"} err="failed to get container status \"9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be\": rpc error: code = NotFound desc = could not find container \"9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be\": container with ID starting with 9ec4dd1fdb6c9166351a75b178dee49637882c2825e135b7b9716f82b372c3be not found: ID does not exist" Oct 30 04:25:45.778596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.778565 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31400a77-11ef-4706-999a-5f60cee5ecf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31400a77-11ef-4706-999a-5f60cee5ecf7-plugins-config-volume\") pod \"31400a77-11ef-4706-999a-5f60cee5ecf7\" (UID: \"31400a77-11ef-4706-999a-5f60cee5ecf7\") " Oct 30 04:25:45.778740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.778715 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31400a77-11ef-4706-999a-5f60cee5ecf7-kube-api-access-9r8zc\" (UniqueName: \"kubernetes.io/projected/31400a77-11ef-4706-999a-5f60cee5ecf7-kube-api-access-9r8zc\") pod \"31400a77-11ef-4706-999a-5f60cee5ecf7\" (UID: \"31400a77-11ef-4706-999a-5f60cee5ecf7\") " Oct 30 04:25:45.778886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.778864 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31400a77-11ef-4706-999a-5f60cee5ecf7-plugins-config-volume" pod "31400a77-11ef-4706-999a-5f60cee5ecf7" (UID: "31400a77-11ef-4706-999a-5f60cee5ecf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:45.779227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.779172 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31400a77-11ef-4706-999a-5f60cee5ecf7-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:45.780786 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.780762 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31400a77-11ef-4706-999a-5f60cee5ecf7-kube-api-access-9r8zc" pod "31400a77-11ef-4706-999a-5f60cee5ecf7" (UID: "31400a77-11ef-4706-999a-5f60cee5ecf7"). InnerVolumeSpecName "kube-api-access-9r8zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:45.880591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:45.880514 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r8zc\" (UniqueName: \"kubernetes.io/projected/31400a77-11ef-4706-999a-5f60cee5ecf7-kube-api-access-9r8zc\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:46.092321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:46.092288 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4"] Oct 30 04:25:46.102615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:46.102584 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e3c43b-644fb88f75-mrvv4"] Oct 30 04:25:46.847615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:46.847584 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31400a77-11ef-4706-999a-5f60cee5ecf7" path="/var/lib/kubelet/pods/31400a77-11ef-4706-999a-5f60cee5ecf7/volumes" Oct 30 04:25:47.064906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:47.064868 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:47.281865 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:47.281829 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:48.430842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:48.430803 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:48.639564 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:48.639516 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:48.839214 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:48.839120 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:49.109711 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:49.109603 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:49.544937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:49.544895 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:50.596941 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:50.596902 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:51.635179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:51.635146 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j"] Oct 30 04:25:51.635568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:51.635349 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" podUID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" containerName="istio-proxy" containerID="cri-o://64dc6156ffd278687bfccbbc5125535aa63bcfbbf49eb70b9c0b8476cbf47b88" gracePeriod=30 Oct 30 04:25:51.874115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:51.874075 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:52.084848 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:52.084811 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:52.085831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:52.085809 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv"] Oct 30 04:25:52.086056 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:52.086036 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" podUID="bc1b3d8c-6a80-400c-aebe-bbab2430d494" containerName="istio-proxy" containerID="cri-o://9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5" gracePeriod=30 Oct 30 04:25:52.090074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:52.090050 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822"] Oct 30 04:25:52.090298 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:52.090263 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" podUID="7a607605-2415-4bd8-a313-47cd6089c8a2" containerName="istio-proxy" containerID="cri-o://d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a" gracePeriod=30 Oct 30 04:25:52.364254 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:52.364163 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:52.795600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:52.795570 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:52.998061 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:52.998011 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:53.210887 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:53.210852 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:53.599242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:53.599149 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:53.801137 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:53.801106 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:54.306545 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:54.306491 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:54.536817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:54.536785 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg"] Oct 30 04:25:54.537079 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:54.537046 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" containerID="cri-o://d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff" gracePeriod=30 Oct 30 04:25:54.662280 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:54.662194 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:54.925442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:54.925385 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:55.414111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:55.414026 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:56.669832 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:56.669802 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:56.845048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:56.844495 4793 generic.go:356] "Generic (PLEG): container finished" podID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" containerID="64dc6156ffd278687bfccbbc5125535aa63bcfbbf49eb70b9c0b8476cbf47b88" exitCode=0 Oct 30 04:25:56.854070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:56.854037 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" event={"ID":"a5a4742f-1003-46d5-bdad-bc7eebe769dc","Type":"ContainerDied","Data":"64dc6156ffd278687bfccbbc5125535aa63bcfbbf49eb70b9c0b8476cbf47b88"} Oct 30 04:25:56.919160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:56.919136 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:25:56.931385 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:56.931349 4793 patch_prober.go:29] interesting pod/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:25:56.931593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:56.931490 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:25:57.071754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.071721 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-podinfo\") pod \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " Oct 30 04:25:57.071915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.071767 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-socket\") pod \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " Oct 30 04:25:57.071915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.071816 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-envoy\") pod \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " Oct 30 04:25:57.071915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.071886 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-credential-socket\") pod \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " Oct 30 04:25:57.072089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.071993 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-kube-api-access-xqfcb\" (UniqueName: \"kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-kube-api-access-xqfcb\") pod \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " Oct 30 04:25:57.072089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.072024 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-token\" (UniqueName: \"kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-token\") pod \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " Oct 30 04:25:57.072213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.072089 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istiod-ca-cert\") pod \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " Oct 30 04:25:57.072213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.072139 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-data\") pod \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " Oct 30 04:25:57.072213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.072140 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-socket" pod "a5a4742f-1003-46d5-bdad-bc7eebe769dc" (UID: "a5a4742f-1003-46d5-bdad-bc7eebe769dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.072362 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.072273 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-certs\") pod \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\" (UID: \"a5a4742f-1003-46d5-bdad-bc7eebe769dc\") " Oct 30 04:25:57.072554 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.072527 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-credential-socket" pod "a5a4742f-1003-46d5-bdad-bc7eebe769dc" (UID: "a5a4742f-1003-46d5-bdad-bc7eebe769dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.072771 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.072745 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-certs" pod "a5a4742f-1003-46d5-bdad-bc7eebe769dc" (UID: "a5a4742f-1003-46d5-bdad-bc7eebe769dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.073028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.072998 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.073028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.073014 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-data" pod "a5a4742f-1003-46d5-bdad-bc7eebe769dc" (UID: "a5a4742f-1003-46d5-bdad-bc7eebe769dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.073175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.073021 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.073175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.073076 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.073315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.072936 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istiod-ca-cert" pod "a5a4742f-1003-46d5-bdad-bc7eebe769dc" (UID: "a5a4742f-1003-46d5-bdad-bc7eebe769dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:57.074508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.074480 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-podinfo" pod "a5a4742f-1003-46d5-bdad-bc7eebe769dc" (UID: "a5a4742f-1003-46d5-bdad-bc7eebe769dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:25:57.074647 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.074619 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-token" pod "a5a4742f-1003-46d5-bdad-bc7eebe769dc" (UID: "a5a4742f-1003-46d5-bdad-bc7eebe769dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:57.075177 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.075151 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-envoy" pod "a5a4742f-1003-46d5-bdad-bc7eebe769dc" (UID: "a5a4742f-1003-46d5-bdad-bc7eebe769dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.075371 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.075350 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-kube-api-access-xqfcb" pod "a5a4742f-1003-46d5-bdad-bc7eebe769dc" (UID: "a5a4742f-1003-46d5-bdad-bc7eebe769dc"). InnerVolumeSpecName "kube-api-access-xqfcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:57.174474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.174448 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.174474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.174473 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.174657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.174482 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqfcb\" (UniqueName: \"kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-kube-api-access-xqfcb\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.174657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.174491 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.174657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.174500 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.174657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.174507 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5a4742f-1003-46d5-bdad-bc7eebe769dc-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.303783 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.303749 4793 scope.go:122] "RemoveContainer" containerID="bb3005d721765549bbb8a72fe67f284e8d59615df6c82f996da6cef4c0ab1bc8" Oct 30 04:25:57.317678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.317650 4793 scope.go:122] "RemoveContainer" containerID="fa060d7faf23e84b114891ae64743bb87c3ddd3de57de7741baa65423b865190" Oct 30 04:25:57.339178 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.339150 4793 scope.go:122] "RemoveContainer" containerID="3d985fdec9a9229f7e486fa0b2bee2e76b991c39d40666189fcf6fe23f8464bf" Oct 30 04:25:57.353140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.353117 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:25:57.355669 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.355650 4793 scope.go:122] "RemoveContainer" containerID="91b20cbe0da32f3ec8e2a8314ab288850ae91537f81692ef999ae90f231bf230" Oct 30 04:25:57.365072 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.365053 4793 scope.go:122] "RemoveContainer" containerID="581f480ed311b6188ad034ac7e793a68a79a9778fb68e6380a8314b46ddf3311" Oct 30 04:25:57.376800 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.376777 4793 scope.go:122] "RemoveContainer" containerID="82d1d02ed13ce249ef6cd3130dcbb1c0da5106c37ef5a742cd68c8f410bc6e95" Oct 30 04:25:57.390411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.390393 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:25:57.478522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.478398 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-envoy\") pod \"7a607605-2415-4bd8-a313-47cd6089c8a2\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " Oct 30 04:25:57.478522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.478506 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-podinfo\") pod \"7a607605-2415-4bd8-a313-47cd6089c8a2\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " Oct 30 04:25:57.478725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.478551 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-certs\") pod \"7a607605-2415-4bd8-a313-47cd6089c8a2\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " Oct 30 04:25:57.478725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.478671 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-credential-socket\") pod \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " Oct 30 04:25:57.478851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.478745 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-data\") pod \"7a607605-2415-4bd8-a313-47cd6089c8a2\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " Oct 30 04:25:57.478851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.478778 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istiod-ca-cert\") pod \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " Oct 30 04:25:57.478851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.478810 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-certs\") pod \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " Oct 30 04:25:57.479004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.478892 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-kube-api-access-lkc7n\" (UniqueName: \"kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-kube-api-access-lkc7n\") pod \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " Oct 30 04:25:57.479004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.478923 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-kube-api-access-f67zh\" (UniqueName: \"kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-kube-api-access-f67zh\") pod \"7a607605-2415-4bd8-a313-47cd6089c8a2\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " Oct 30 04:25:57.479004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.478953 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-token\" (UniqueName: \"kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-token\") pod \"7a607605-2415-4bd8-a313-47cd6089c8a2\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " Oct 30 04:25:57.479004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.478992 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a607605-2415-4bd8-a313-47cd6089c8a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a607605-2415-4bd8-a313-47cd6089c8a2-istiod-ca-cert\") pod \"7a607605-2415-4bd8-a313-47cd6089c8a2\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " Oct 30 04:25:57.479263 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.479037 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-socket\") pod \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " Oct 30 04:25:57.479263 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.479080 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-token\" (UniqueName: \"kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-token\") pod \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " Oct 30 04:25:57.479263 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.479156 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-podinfo\") pod \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " Oct 30 04:25:57.479263 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.479191 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-envoy\") pod \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " Oct 30 04:25:57.479263 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.479231 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-data\") pod \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\" (UID: \"bc1b3d8c-6a80-400c-aebe-bbab2430d494\") " Oct 30 04:25:57.479263 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.479260 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-socket\") pod \"7a607605-2415-4bd8-a313-47cd6089c8a2\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " Oct 30 04:25:57.479615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.479294 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-credential-socket" pod "bc1b3d8c-6a80-400c-aebe-bbab2430d494" (UID: "bc1b3d8c-6a80-400c-aebe-bbab2430d494"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.479615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.479317 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-credential-socket\") pod \"7a607605-2415-4bd8-a313-47cd6089c8a2\" (UID: \"7a607605-2415-4bd8-a313-47cd6089c8a2\") " Oct 30 04:25:57.480189 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.479934 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.480189 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.480037 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-certs" pod "bc1b3d8c-6a80-400c-aebe-bbab2430d494" (UID: "bc1b3d8c-6a80-400c-aebe-bbab2430d494"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.480189 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.480062 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a607605-2415-4bd8-a313-47cd6089c8a2-istiod-ca-cert" pod "7a607605-2415-4bd8-a313-47cd6089c8a2" (UID: "7a607605-2415-4bd8-a313-47cd6089c8a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:57.480189 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.480170 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-credential-socket" pod "7a607605-2415-4bd8-a313-47cd6089c8a2" (UID: "7a607605-2415-4bd8-a313-47cd6089c8a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.480470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.480307 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-certs" pod "7a607605-2415-4bd8-a313-47cd6089c8a2" (UID: "7a607605-2415-4bd8-a313-47cd6089c8a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.480540 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.480467 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istiod-ca-cert" pod "bc1b3d8c-6a80-400c-aebe-bbab2430d494" (UID: "bc1b3d8c-6a80-400c-aebe-bbab2430d494"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:57.482234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.481359 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-data" pod "bc1b3d8c-6a80-400c-aebe-bbab2430d494" (UID: "bc1b3d8c-6a80-400c-aebe-bbab2430d494"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.482234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.481535 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-data" pod "7a607605-2415-4bd8-a313-47cd6089c8a2" (UID: "7a607605-2415-4bd8-a313-47cd6089c8a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.482234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.481600 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-socket" pod "7a607605-2415-4bd8-a313-47cd6089c8a2" (UID: "7a607605-2415-4bd8-a313-47cd6089c8a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.482234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.481787 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-socket" pod "bc1b3d8c-6a80-400c-aebe-bbab2430d494" (UID: "bc1b3d8c-6a80-400c-aebe-bbab2430d494"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.486491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.484218 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-podinfo" pod "bc1b3d8c-6a80-400c-aebe-bbab2430d494" (UID: "bc1b3d8c-6a80-400c-aebe-bbab2430d494"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:25:57.486491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.484870 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-kube-api-access-f67zh" pod "7a607605-2415-4bd8-a313-47cd6089c8a2" (UID: "7a607605-2415-4bd8-a313-47cd6089c8a2"). InnerVolumeSpecName "kube-api-access-f67zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:57.486491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.485491 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-envoy" pod "7a607605-2415-4bd8-a313-47cd6089c8a2" (UID: "7a607605-2415-4bd8-a313-47cd6089c8a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.489539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.489515 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-kube-api-access-lkc7n" pod "bc1b3d8c-6a80-400c-aebe-bbab2430d494" (UID: "bc1b3d8c-6a80-400c-aebe-bbab2430d494"). InnerVolumeSpecName "kube-api-access-lkc7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:57.489624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.489582 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-podinfo" pod "7a607605-2415-4bd8-a313-47cd6089c8a2" (UID: "7a607605-2415-4bd8-a313-47cd6089c8a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:25:57.489624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.489611 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-envoy" pod "bc1b3d8c-6a80-400c-aebe-bbab2430d494" (UID: "bc1b3d8c-6a80-400c-aebe-bbab2430d494"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:57.489736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.489618 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-token" pod "7a607605-2415-4bd8-a313-47cd6089c8a2" (UID: "7a607605-2415-4bd8-a313-47cd6089c8a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:57.489736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.489644 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-token" pod "bc1b3d8c-6a80-400c-aebe-bbab2430d494" (UID: "bc1b3d8c-6a80-400c-aebe-bbab2430d494"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:57.581114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581085 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581112 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581122 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581131 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581140 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581148 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581156 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581164 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581172 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581185 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581196 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkc7n\" (UniqueName: \"kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-kube-api-access-lkc7n\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581205 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f67zh\" (UniqueName: \"kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-kube-api-access-f67zh\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581212 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a607605-2415-4bd8-a313-47cd6089c8a2-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581222 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a607605-2415-4bd8-a313-47cd6089c8a2-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581230 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1b3d8c-6a80-400c-aebe-bbab2430d494-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581237 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.581352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.581245 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1b3d8c-6a80-400c-aebe-bbab2430d494-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:57.852449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.852407 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" Oct 30 04:25:57.852449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.852428 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j" event={"ID":"a5a4742f-1003-46d5-bdad-bc7eebe769dc","Type":"ContainerDied","Data":"de9140c4cd92b582bc741676179d35b82cf7647da5d533ded71955bad6c16cef"} Oct 30 04:25:57.852983 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.852486 4793 scope.go:122] "RemoveContainer" containerID="64dc6156ffd278687bfccbbc5125535aa63bcfbbf49eb70b9c0b8476cbf47b88" Oct 30 04:25:57.853901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.853880 4793 generic.go:356] "Generic (PLEG): container finished" podID="7a607605-2415-4bd8-a313-47cd6089c8a2" containerID="d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a" exitCode=0 Oct 30 04:25:57.853993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.853933 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" Oct 30 04:25:57.854048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.853942 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" event={"ID":"7a607605-2415-4bd8-a313-47cd6089c8a2","Type":"ContainerDied","Data":"d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a"} Oct 30 04:25:57.854096 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.854054 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822" event={"ID":"7a607605-2415-4bd8-a313-47cd6089c8a2","Type":"ContainerDied","Data":"81e7529577319a8528b4ca423e2b24d8273e3f9b023b397c0b312fd787de991e"} Oct 30 04:25:57.858431 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.858391 4793 generic.go:356] "Generic (PLEG): container finished" podID="bc1b3d8c-6a80-400c-aebe-bbab2430d494" containerID="9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5" exitCode=0 Oct 30 04:25:57.858529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.858469 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" event={"ID":"bc1b3d8c-6a80-400c-aebe-bbab2430d494","Type":"ContainerDied","Data":"9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5"} Oct 30 04:25:57.858529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.858512 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" event={"ID":"bc1b3d8c-6a80-400c-aebe-bbab2430d494","Type":"ContainerDied","Data":"32be328ab27b8e60ec735af36eeab7f4609f2ceb5e04207d6c8ce315bbaee3d1"} Oct 30 04:25:57.858631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.858472 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv" Oct 30 04:25:57.865815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.865798 4793 scope.go:122] "RemoveContainer" containerID="d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a" Oct 30 04:25:57.874747 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.874730 4793 scope.go:122] "RemoveContainer" containerID="d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a" Oct 30 04:25:57.874989 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:25:57.874974 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a\": container with ID starting with d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a not found: ID does not exist" containerID="d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a" Oct 30 04:25:57.875032 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.874998 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a"} err="failed to get container status \"d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a\": rpc error: code = NotFound desc = could not find container \"d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a\": container with ID starting with d1298511bc182a4f7e8898825c84fcfd3778ac0dbdf4e3c11f9a2d78fe57e72a not found: ID does not exist" Oct 30 04:25:57.875032 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.875012 4793 scope.go:122] "RemoveContainer" containerID="9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5" Oct 30 04:25:57.883566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.883552 4793 scope.go:122] "RemoveContainer" containerID="9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5" Oct 30 04:25:57.883788 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:25:57.883771 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5\": container with ID starting with 9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5 not found: ID does not exist" containerID="9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5" Oct 30 04:25:57.883832 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.883792 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5"} err="failed to get container status \"9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5\": rpc error: code = NotFound desc = could not find container \"9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5\": container with ID starting with 9100b23cf6abcf43fbdfc551dcc14f36d1c32fc47baae6144eeb01ed9c0ed0b5 not found: ID does not exist" Oct 30 04:25:57.913145 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.913122 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j"] Oct 30 04:25:57.925205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.925185 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-z5n8j"] Oct 30 04:25:57.951717 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.951695 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv"] Oct 30 04:25:57.958673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.958645 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-v9qlv"] Oct 30 04:25:57.992658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.992636 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822"] Oct 30 04:25:57.997031 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:57.997010 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-2z822"] Oct 30 04:25:58.382823 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:58.382783 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:58.583366 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:58.583328 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:58.780627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:58.780579 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:25:58.847686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:58.847657 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a607605-2415-4bd8-a313-47cd6089c8a2" path="/var/lib/kubelet/pods/7a607605-2415-4bd8-a313-47cd6089c8a2/volumes" Oct 30 04:25:58.848117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:58.848103 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" path="/var/lib/kubelet/pods/a5a4742f-1003-46d5-bdad-bc7eebe769dc/volumes" Oct 30 04:25:58.848566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:58.848551 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1b3d8c-6a80-400c-aebe-bbab2430d494" path="/var/lib/kubelet/pods/bc1b3d8c-6a80-400c-aebe-bbab2430d494/volumes" Oct 30 04:25:59.827206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.827182 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:25:59.882138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.882098 4793 generic.go:356] "Generic (PLEG): container finished" podID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerID="d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff" exitCode=0 Oct 30 04:25:59.882391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.882157 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" event={"ID":"1d17eab8-a71b-4fc5-b66b-e8026f8f874c","Type":"ContainerDied","Data":"d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff"} Oct 30 04:25:59.882583 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.882447 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" event={"ID":"1d17eab8-a71b-4fc5-b66b-e8026f8f874c","Type":"ContainerDied","Data":"7342b6fd453698195595d95e3c0d866c1986c9774dbf5f44775bee23895fefab"} Oct 30 04:25:59.882583 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.882483 4793 scope.go:122] "RemoveContainer" containerID="d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff" Oct 30 04:25:59.882583 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.882258 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg" Oct 30 04:25:59.916258 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.916226 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-token\" (UniqueName: \"kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-token\") pod \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " Oct 30 04:25:59.916536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.916513 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-podinfo\") pod \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " Oct 30 04:25:59.917184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.916718 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-socket\") pod \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " Oct 30 04:25:59.917391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.917370 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-certs\") pod \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " Oct 30 04:25:59.917605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.917582 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istiod-ca-cert\") pod \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " Oct 30 04:25:59.917734 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.917716 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-credential-socket\") pod \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " Oct 30 04:25:59.917884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.917868 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-kube-api-access-67wsw\" (UniqueName: \"kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-kube-api-access-67wsw\") pod \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " Oct 30 04:25:59.918065 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.918035 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-socket" pod "1d17eab8-a71b-4fc5-b66b-e8026f8f874c" (UID: "1d17eab8-a71b-4fc5-b66b-e8026f8f874c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:59.918211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.918193 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-data\") pod \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " Oct 30 04:25:59.918412 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.918395 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-envoy\") pod \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\" (UID: \"1d17eab8-a71b-4fc5-b66b-e8026f8f874c\") " Oct 30 04:25:59.918573 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.917776 4793 scope.go:122] "RemoveContainer" containerID="9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae" Oct 30 04:25:59.918899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.918871 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-data" pod "1d17eab8-a71b-4fc5-b66b-e8026f8f874c" (UID: "1d17eab8-a71b-4fc5-b66b-e8026f8f874c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:59.919427 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.919056 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-credential-socket" pod "1d17eab8-a71b-4fc5-b66b-e8026f8f874c" (UID: "1d17eab8-a71b-4fc5-b66b-e8026f8f874c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:59.919427 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.919198 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:59.919427 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.919214 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:59.919427 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.919223 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:25:59.919427 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.919386 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istiod-ca-cert" pod "1d17eab8-a71b-4fc5-b66b-e8026f8f874c" (UID: "1d17eab8-a71b-4fc5-b66b-e8026f8f874c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:25:59.921052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.921018 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-certs" pod "1d17eab8-a71b-4fc5-b66b-e8026f8f874c" (UID: "1d17eab8-a71b-4fc5-b66b-e8026f8f874c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:59.924202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.924180 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-kube-api-access-67wsw" pod "1d17eab8-a71b-4fc5-b66b-e8026f8f874c" (UID: "1d17eab8-a71b-4fc5-b66b-e8026f8f874c"). InnerVolumeSpecName "kube-api-access-67wsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:59.926900 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.926867 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-token" pod "1d17eab8-a71b-4fc5-b66b-e8026f8f874c" (UID: "1d17eab8-a71b-4fc5-b66b-e8026f8f874c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:25:59.927761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.927739 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-podinfo" pod "1d17eab8-a71b-4fc5-b66b-e8026f8f874c" (UID: "1d17eab8-a71b-4fc5-b66b-e8026f8f874c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:25:59.929025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.928990 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-envoy" pod "1d17eab8-a71b-4fc5-b66b-e8026f8f874c" (UID: "1d17eab8-a71b-4fc5-b66b-e8026f8f874c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:25:59.957454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.957434 4793 scope.go:122] "RemoveContainer" containerID="d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff" Oct 30 04:25:59.957800 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:25:59.957776 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff\": container with ID starting with d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff not found: ID does not exist" containerID="d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff" Oct 30 04:25:59.957904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.957807 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff"} err="failed to get container status \"d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff\": rpc error: code = NotFound desc = could not find container \"d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff\": container with ID starting with d0c6f87ca44572eb5dc71a86981c9d1c271048aa877685f80bb40375549459ff not found: ID does not exist" Oct 30 04:25:59.957904 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.957823 4793 scope.go:122] "RemoveContainer" containerID="9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae" Oct 30 04:25:59.958202 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:25:59.958070 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae\": container with ID starting with 9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae not found: ID does not exist" containerID="9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae" Oct 30 04:25:59.958202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:25:59.958094 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae"} err="failed to get container status \"9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae\": rpc error: code = NotFound desc = could not find container \"9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae\": container with ID starting with 9be634242d14b1d93ba8a5fb2f6033c3edb03f68ce96ffe7c63941f0c70e91ae not found: ID does not exist" Oct 30 04:26:00.020695 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:00.020670 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:00.020695 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:00.020692 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:00.020864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:00.020703 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67wsw\" (UniqueName: \"kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-kube-api-access-67wsw\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:00.020864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:00.020714 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:00.020864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:00.020723 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:00.020864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:00.020733 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d17eab8-a71b-4fc5-b66b-e8026f8f874c-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:00.144720 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:00.144625 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:00.233642 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:00.233609 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg"] Oct 30 04:26:00.237286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:00.237260 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e3c43b-openshift-default-86b776b77c-wmtkg"] Oct 30 04:26:00.382770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:00.382735 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:00.846842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:00.846812 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" path="/var/lib/kubelet/pods/1d17eab8-a71b-4fc5-b66b-e8026f8f874c/volumes" Oct 30 04:26:00.944524 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:00.944493 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm"] Oct 30 04:26:01.137463 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:01.137368 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:01.467724 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:01.467694 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:02.012480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:02.012440 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:02.359797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:02.359705 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:03.913073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:03.913033 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:05.810471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:05.801067 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:05.945380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:05.945342 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" podUID="c4d55806-458c-454d-998f-f56e3eb4363f" containerName="payload-pre-processing" containerID="cri-o://296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1" gracePeriod=25 Oct 30 04:26:06.226266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.226237 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" Oct 30 04:26:06.239787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.239759 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d55806-458c-454d-998f-f56e3eb4363f-kube-api-access-qml8t\" (UniqueName: \"kubernetes.io/projected/c4d55806-458c-454d-998f-f56e3eb4363f-kube-api-access-qml8t\") pod \"c4d55806-458c-454d-998f-f56e3eb4363f\" (UID: \"c4d55806-458c-454d-998f-f56e3eb4363f\") " Oct 30 04:26:06.239894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.239824 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d55806-458c-454d-998f-f56e3eb4363f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d55806-458c-454d-998f-f56e3eb4363f-plugins-config-volume\") pod \"c4d55806-458c-454d-998f-f56e3eb4363f\" (UID: \"c4d55806-458c-454d-998f-f56e3eb4363f\") " Oct 30 04:26:06.240181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.240156 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d55806-458c-454d-998f-f56e3eb4363f-plugins-config-volume" pod "c4d55806-458c-454d-998f-f56e3eb4363f" (UID: "c4d55806-458c-454d-998f-f56e3eb4363f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:26:06.240279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.240266 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d55806-458c-454d-998f-f56e3eb4363f-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:06.241905 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.241885 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d55806-458c-454d-998f-f56e3eb4363f-kube-api-access-qml8t" pod "c4d55806-458c-454d-998f-f56e3eb4363f" (UID: "c4d55806-458c-454d-998f-f56e3eb4363f"). InnerVolumeSpecName "kube-api-access-qml8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:26:06.340993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.340906 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qml8t\" (UniqueName: \"kubernetes.io/projected/c4d55806-458c-454d-998f-f56e3eb4363f-kube-api-access-qml8t\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:06.946001 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.945971 4793 generic.go:356] "Generic (PLEG): container finished" podID="c4d55806-458c-454d-998f-f56e3eb4363f" containerID="296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1" exitCode=0 Oct 30 04:26:06.946400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.946024 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" event={"ID":"c4d55806-458c-454d-998f-f56e3eb4363f","Type":"ContainerDied","Data":"296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1"} Oct 30 04:26:06.946400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.946031 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" Oct 30 04:26:06.946400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.946063 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" event={"ID":"c4d55806-458c-454d-998f-f56e3eb4363f","Type":"ContainerDied","Data":"85c2be508a67295361dc570416e7f4855ff2126c02f4864e82f15f64e46d7359"} Oct 30 04:26:06.946400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.946084 4793 scope.go:122] "RemoveContainer" containerID="296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1" Oct 30 04:26:06.956770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.956746 4793 scope.go:122] "RemoveContainer" containerID="296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1" Oct 30 04:26:06.957117 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:26:06.957094 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1\": container with ID starting with 296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1 not found: ID does not exist" containerID="296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1" Oct 30 04:26:06.957193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:06.957125 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1"} err="failed to get container status \"296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1\": rpc error: code = NotFound desc = could not find container \"296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1\": container with ID starting with 296b6c9be912d2354f29cda2e4e60864b624134efb0a4851e6bf81bc7c2d20f1 not found: ID does not exist" Oct 30 04:26:07.002785 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:07.002755 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm"] Oct 30 04:26:07.006627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:07.006605 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm"] Oct 30 04:26:07.121657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:07.121567 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-397bda-86b6678695-xb5wm" podUID="c4d55806-458c-454d-998f-f56e3eb4363f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.107:50052\" within 1s: context deadline exceeded" Oct 30 04:26:07.121657 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:26:07.121623 4793 logging.go:55] [core] [Channel #1718 SubChannel #1719]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.107:50052", ServerName: "10.133.0.107:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.107:50052: operation was canceled" Oct 30 04:26:07.205601 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:07.205564 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:07.415358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:07.415275 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:08.230696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:08.230661 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:08.822193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:08.822142 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:08.846940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:08.846912 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d55806-458c-454d-998f-f56e3eb4363f" path="/var/lib/kubelet/pods/c4d55806-458c-454d-998f-f56e3eb4363f/volumes" Oct 30 04:26:09.103520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.103404 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:09.133470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.133438 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25"] Oct 30 04:26:09.134046 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134027 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1d74fd83-dd2d-4b3e-8269-b96ff52ed274" containerName="payload-pre-processing" Oct 30 04:26:09.134143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134047 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d74fd83-dd2d-4b3e-8269-b96ff52ed274" containerName="payload-pre-processing" Oct 30 04:26:09.134143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134069 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" containerName="istio-proxy" Oct 30 04:26:09.134143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134079 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" containerName="istio-proxy" Oct 30 04:26:09.134143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134094 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" Oct 30 04:26:09.134143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134101 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" Oct 30 04:26:09.134143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134114 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" Oct 30 04:26:09.134143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134121 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" Oct 30 04:26:09.134143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134134 4793 cpu_manager.go:414] "containerMap: removing container" podUID="31400a77-11ef-4706-999a-5f60cee5ecf7" containerName="payload-pre-processing" Oct 30 04:26:09.134143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134141 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31400a77-11ef-4706-999a-5f60cee5ecf7" containerName="payload-pre-processing" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134152 4793 cpu_manager.go:414] "containerMap: removing container" podUID="025f0ebe-c1e2-439f-8072-8543c281269a" containerName="payload-processing" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134159 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="025f0ebe-c1e2-439f-8072-8543c281269a" containerName="payload-processing" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134172 4793 cpu_manager.go:414] "containerMap: removing container" podUID="bc1b3d8c-6a80-400c-aebe-bbab2430d494" containerName="istio-proxy" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134181 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1b3d8c-6a80-400c-aebe-bbab2430d494" containerName="istio-proxy" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134212 4793 cpu_manager.go:414] "containerMap: removing container" podUID="379aab4d-7a33-4ca0-bbd7-8ffb091638d5" containerName="revoke-keys" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134220 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379aab4d-7a33-4ca0-bbd7-8ffb091638d5" containerName="revoke-keys" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134238 4793 cpu_manager.go:414] "containerMap: removing container" podUID="7a607605-2415-4bd8-a313-47cd6089c8a2" containerName="istio-proxy" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134245 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a607605-2415-4bd8-a313-47cd6089c8a2" containerName="istio-proxy" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134257 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d9857c7f-c96b-4057-8c34-7359cf82ffef" containerName="payload-processing" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134264 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9857c7f-c96b-4057-8c34-7359cf82ffef" containerName="payload-processing" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134275 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c4d55806-458c-454d-998f-f56e3eb4363f" containerName="payload-pre-processing" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134284 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d55806-458c-454d-998f-f56e3eb4363f" containerName="payload-pre-processing" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134455 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="31400a77-11ef-4706-999a-5f60cee5ecf7" containerName="payload-pre-processing" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134474 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="025f0ebe-c1e2-439f-8072-8543c281269a" containerName="payload-processing" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134486 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1b3d8c-6a80-400c-aebe-bbab2430d494" containerName="istio-proxy" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134501 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="379aab4d-7a33-4ca0-bbd7-8ffb091638d5" containerName="revoke-keys" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134512 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a607605-2415-4bd8-a313-47cd6089c8a2" containerName="istio-proxy" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134520 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9857c7f-c96b-4057-8c34-7359cf82ffef" containerName="payload-processing" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134529 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d55806-458c-454d-998f-f56e3eb4363f" containerName="payload-pre-processing" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134543 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d74fd83-dd2d-4b3e-8269-b96ff52ed274" containerName="payload-pre-processing" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134554 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134564 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a4742f-1003-46d5-bdad-bc7eebe769dc" containerName="istio-proxy" Oct 30 04:26:09.134606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.134573 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d17eab8-a71b-4fc5-b66b-e8026f8f874c" containerName="istio-proxy" Oct 30 04:26:09.137994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.137970 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.163269 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.163190 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bf4238-openshift-default-dockercfg-8gw8k\"" Oct 30 04:26:09.177454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.176188 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25"] Oct 30 04:26:09.177454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.176736 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62df4ee-f0ca-4788-8723-1619ec96c657-istiod-ca-cert\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.177454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.176772 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-socket\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.177454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.176819 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-data\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.177454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.176858 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-podinfo\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.177454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.176876 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-envoy\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.177454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.176924 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79cq7\" (UniqueName: \"kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-kube-api-access-79cq7\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.177454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.176953 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-credential-socket\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.177454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.177009 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-token\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.177454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.177031 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-certs\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.278516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.278482 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79cq7\" (UniqueName: \"kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-kube-api-access-79cq7\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.278994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.278538 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-credential-socket\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.278994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.278601 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-token\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.278994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.278636 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-certs\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.278994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.278727 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62df4ee-f0ca-4788-8723-1619ec96c657-istiod-ca-cert\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.278994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.278832 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-socket\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.278994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.278903 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-data\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.278994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.278950 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-podinfo\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.278994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.278978 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-envoy\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.280099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.279043 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-credential-socket\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.280099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.279095 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-certs\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.280099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.279353 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-socket\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.280099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.279498 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-data\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.280099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.279580 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62df4ee-f0ca-4788-8723-1619ec96c657-istiod-ca-cert\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.281491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.281467 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-envoy\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.281768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.281750 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-podinfo\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.304999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.304970 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79cq7\" (UniqueName: \"kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-kube-api-access-79cq7\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.305206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.305189 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-token\") pod \"e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.307654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.307628 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:09.459052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.459009 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:09.513912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.513883 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:09.784363 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:26:09.784338 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf62df4ee_f0ca_4788_8723_1619ec96c657.slice/crio-65369617012c0ca8aaa75a1ea4c4c0034441582baa4ba96652792f25172325b6 WatchSource:0}: Error finding container 65369617012c0ca8aaa75a1ea4c4c0034441582baa4ba96652792f25172325b6: Status 404 returned error can't find the container with id 65369617012c0ca8aaa75a1ea4c4c0034441582baa4ba96652792f25172325b6 Oct 30 04:26:09.784480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.784451 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25"] Oct 30 04:26:09.786829 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.786796 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:26:09.786934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.786873 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:26:09.786934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.786916 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:26:09.971955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.971921 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" event={"ID":"f62df4ee-f0ca-4788-8723-1619ec96c657","Type":"ContainerStarted","Data":"bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98"} Oct 30 04:26:09.972141 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:09.971977 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" event={"ID":"f62df4ee-f0ca-4788-8723-1619ec96c657","Type":"ContainerStarted","Data":"65369617012c0ca8aaa75a1ea4c4c0034441582baa4ba96652792f25172325b6"} Oct 30 04:26:10.018868 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:10.018818 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" podStartSLOduration=1.018803588 podStartE2EDuration="1.018803588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:10.017292527 +0000 UTC m=+1693.303167303" watchObservedRunningTime="2026-10-30 04:26:10.018803588 +0000 UTC m=+1693.304678362" Oct 30 04:26:10.459857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:10.459822 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:10.464757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:10.464732 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:10.549820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:10.549775 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:10.987171 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:10.985475 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:10.989570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:10.989252 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:11.207460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:11.207410 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:11.227634 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:26:11.227599 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:26:12.437900 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:12.437860 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:12.664478 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:12.661591 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:13.198377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:13.198336 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:13.597430 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:13.597328 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:14.196775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:14.196583 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:14.419307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:14.419271 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:14.701789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:14.701754 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:14.794142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:14.794114 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm"] Oct 30 04:26:14.797083 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:14.797064 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" Oct 30 04:26:14.816440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:14.816407 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm"] Oct 30 04:26:14.824652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:14.824629 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf4238\"" Oct 30 04:26:14.824785 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:14.824735 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf4238-dockercfg-lwx48\"" Oct 30 04:26:14.900439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:14.900129 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rjj\" (UniqueName: \"kubernetes.io/projected/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-kube-api-access-w9rjj\") pod \"payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm\" (UID: \"cb4fb7a2-740e-4ccf-8255-557aeb555b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" Oct 30 04:26:14.952650 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:14.947265 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:14.952650 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:14.948075 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm\" (UID: \"cb4fb7a2-740e-4ccf-8255-557aeb555b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" Oct 30 04:26:15.006247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.006180 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx"] Oct 30 04:26:15.009725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.009700 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" Oct 30 04:26:15.012681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.011987 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx"] Oct 30 04:26:15.049199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.049132 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5sr5\" (UniqueName: \"kubernetes.io/projected/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-kube-api-access-x5sr5\") pod \"payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx\" (UID: \"c5c35ef0-7aa2-4a52-9463-800df5b6cd19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" Oct 30 04:26:15.049327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.049242 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx\" (UID: \"c5c35ef0-7aa2-4a52-9463-800df5b6cd19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" Oct 30 04:26:15.049327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.049294 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm\" (UID: \"cb4fb7a2-740e-4ccf-8255-557aeb555b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" Oct 30 04:26:15.049494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.049460 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rjj\" (UniqueName: \"kubernetes.io/projected/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-kube-api-access-w9rjj\") pod \"payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm\" (UID: \"cb4fb7a2-740e-4ccf-8255-557aeb555b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" Oct 30 04:26:15.051594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.051568 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm\" (UID: \"cb4fb7a2-740e-4ccf-8255-557aeb555b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" Oct 30 04:26:15.073679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.073649 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rjj\" (UniqueName: \"kubernetes.io/projected/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-kube-api-access-w9rjj\") pod \"payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm\" (UID: \"cb4fb7a2-740e-4ccf-8255-557aeb555b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" Oct 30 04:26:15.111885 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.111861 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" Oct 30 04:26:15.150466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.150439 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5sr5\" (UniqueName: \"kubernetes.io/projected/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-kube-api-access-x5sr5\") pod \"payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx\" (UID: \"c5c35ef0-7aa2-4a52-9463-800df5b6cd19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" Oct 30 04:26:15.150603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.150500 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx\" (UID: \"c5c35ef0-7aa2-4a52-9463-800df5b6cd19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" Oct 30 04:26:15.151166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.151145 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx\" (UID: \"c5c35ef0-7aa2-4a52-9463-800df5b6cd19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" Oct 30 04:26:15.175357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.175327 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5sr5\" (UniqueName: \"kubernetes.io/projected/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-kube-api-access-x5sr5\") pod \"payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx\" (UID: \"c5c35ef0-7aa2-4a52-9463-800df5b6cd19\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" Oct 30 04:26:15.237954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.237696 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:15.325777 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.325746 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" Oct 30 04:26:15.443008 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:26:15.442976 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4fb7a2_740e_4ccf_8255_557aeb555b8d.slice/crio-0d07de0ecd413ba771c677ea84c172175c943e1bb7fd1bfe073dbb1b078c62cb WatchSource:0}: Error finding container 0d07de0ecd413ba771c677ea84c172175c943e1bb7fd1bfe073dbb1b078c62cb: Status 404 returned error can't find the container with id 0d07de0ecd413ba771c677ea84c172175c943e1bb7fd1bfe073dbb1b078c62cb Oct 30 04:26:15.444818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.444787 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm"] Oct 30 04:26:15.629749 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:26:15.629721 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c35ef0_7aa2_4a52_9463_800df5b6cd19.slice/crio-c2f2126db93e460ef6ebfe6169c2368b2da697a191b422c14b92bb506d324185 WatchSource:0}: Error finding container c2f2126db93e460ef6ebfe6169c2368b2da697a191b422c14b92bb506d324185: Status 404 returned error can't find the container with id c2f2126db93e460ef6ebfe6169c2368b2da697a191b422c14b92bb506d324185 Oct 30 04:26:15.630753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.630732 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx"] Oct 30 04:26:15.923614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:15.923509 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:16.081812 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:16.081771 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" event={"ID":"c5c35ef0-7aa2-4a52-9463-800df5b6cd19","Type":"ContainerStarted","Data":"76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2"} Oct 30 04:26:16.081999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:16.081838 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" event={"ID":"c5c35ef0-7aa2-4a52-9463-800df5b6cd19","Type":"ContainerStarted","Data":"c2f2126db93e460ef6ebfe6169c2368b2da697a191b422c14b92bb506d324185"} Oct 30 04:26:16.082659 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:16.082635 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" Oct 30 04:26:16.084400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:16.084374 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" event={"ID":"cb4fb7a2-740e-4ccf-8255-557aeb555b8d","Type":"ContainerStarted","Data":"e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f"} Oct 30 04:26:16.084525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:16.084436 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" event={"ID":"cb4fb7a2-740e-4ccf-8255-557aeb555b8d","Type":"ContainerStarted","Data":"0d07de0ecd413ba771c677ea84c172175c943e1bb7fd1bfe073dbb1b078c62cb"} Oct 30 04:26:16.084574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:16.084528 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" Oct 30 04:26:16.125052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:16.125004 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" podStartSLOduration=2.124988407 podStartE2EDuration="2.124988407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:16.124162602 +0000 UTC m=+1699.410037385" watchObservedRunningTime="2026-10-30 04:26:16.124988407 +0000 UTC m=+1699.410863182" Oct 30 04:26:16.133448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:16.133400 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:16.167646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:16.167598 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" podStartSLOduration=2.167580568 podStartE2EDuration="2.167580568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:16.164668767 +0000 UTC m=+1699.450543582" watchObservedRunningTime="2026-10-30 04:26:16.167580568 +0000 UTC m=+1699.453455346" Oct 30 04:26:16.711294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:16.711260 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:16.932668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:16.932630 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5"] Oct 30 04:26:16.933098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:16.932890 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" containerID="cri-o://073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6" gracePeriod=30 Oct 30 04:26:17.159922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:17.159827 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:17.391802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:17.391763 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:18.374893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:18.374859 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:18.703832 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:18.703795 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:19.056014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.055929 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:19.295182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.295134 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:19.393219 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.393129 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc"] Oct 30 04:26:19.397988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.397966 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.417059 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.417034 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc"] Oct 30 04:26:19.418453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.418432 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-baa474-openshift-default-dockercfg-w6d7d\"" Oct 30 04:26:19.545449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.543165 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-podinfo\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.545449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.543279 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-envoy\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.545449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.543326 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-token\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.545449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.543444 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-credential-socket\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.545449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.543521 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-socket\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.545449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.543590 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-certs\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.545449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.543616 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc09d10-3490-43e5-9d1f-f52fed67d037-istiod-ca-cert\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.545449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.543641 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-data\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.545449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.543677 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttckx\" (UniqueName: \"kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-kube-api-access-ttckx\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.563809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.563776 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:19.645508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.645324 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-envoy\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.645508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.645390 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-token\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.645805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.645532 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-credential-socket\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.645805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.645653 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-socket\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.645805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.645727 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-certs\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.645805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.645752 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc09d10-3490-43e5-9d1f-f52fed67d037-istiod-ca-cert\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.645805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.645775 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-data\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.646105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.645814 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttckx\" (UniqueName: \"kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-kube-api-access-ttckx\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.646105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.645852 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-podinfo\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.646787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.646759 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-socket\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.650705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.650144 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc09d10-3490-43e5-9d1f-f52fed67d037-istiod-ca-cert\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.650705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.650387 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-certs\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.650705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.650663 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-data\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.651104 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.651066 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-podinfo\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.651208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.651172 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-credential-socket\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.651840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.651806 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-envoy\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.679215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.679187 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-token\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.680878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.680850 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttckx\" (UniqueName: \"kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-kube-api-access-ttckx\") pod \"e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.715845 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.715821 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:19.798245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:19.797093 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:20.036025 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:26:20.035997 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc09d10_3490_43e5_9d1f_f52fed67d037.slice/crio-f0f92ac4320054bb3861b7e43dcb3dedb7d076de889cf3e797a27ba45d02100e WatchSource:0}: Error finding container f0f92ac4320054bb3861b7e43dcb3dedb7d076de889cf3e797a27ba45d02100e: Status 404 returned error can't find the container with id f0f92ac4320054bb3861b7e43dcb3dedb7d076de889cf3e797a27ba45d02100e Oct 30 04:26:20.037974 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.037578 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc"] Oct 30 04:26:20.038671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.038639 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:26:20.038753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.038705 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:26:20.038753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.038736 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:26:20.207520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.207466 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" event={"ID":"5dc09d10-3490-43e5-9d1f-f52fed67d037","Type":"ContainerStarted","Data":"32ece55d2c07dfd9eced0b507e04ad174c042f5d548e7a7a7b8acd293be1ba5a"} Oct 30 04:26:20.207709 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.207540 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" event={"ID":"5dc09d10-3490-43e5-9d1f-f52fed67d037","Type":"ContainerStarted","Data":"f0f92ac4320054bb3861b7e43dcb3dedb7d076de889cf3e797a27ba45d02100e"} Oct 30 04:26:20.213919 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.213886 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:20.278734 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.278683 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" podStartSLOduration=1.278664789 podStartE2EDuration="1.278664789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:20.274451158 +0000 UTC m=+1703.560325956" watchObservedRunningTime="2026-10-30 04:26:20.278664789 +0000 UTC m=+1703.564539564" Oct 30 04:26:20.391563 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.390794 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r"] Oct 30 04:26:20.608352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.608269 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" Oct 30 04:26:20.628816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.628773 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6dcea5-dockercfg-4579v\"" Oct 30 04:26:20.629435 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.629395 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6dcea5\"" Oct 30 04:26:20.631894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.629932 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r"] Oct 30 04:26:20.631894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.630992 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt"] Oct 30 04:26:20.637701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.637680 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" Oct 30 04:26:20.639351 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.639327 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt"] Oct 30 04:26:20.660472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.651884 4793 patch_prober.go:29] interesting pod/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:26:20.660472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.651938 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:26:20.676478 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.676171 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:20.716860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.716829 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:20.777777 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.777738 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svftc\" (UniqueName: \"kubernetes.io/projected/961d5aff-b1b9-4605-806a-7837bdb324a7-kube-api-access-svftc\") pod \"payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r\" (UID: \"961d5aff-b1b9-4605-806a-7837bdb324a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" Oct 30 04:26:20.777954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.777797 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcae537d-1ab4-4ab4-b68d-08bd1000c598-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt\" (UID: \"fcae537d-1ab4-4ab4-b68d-08bd1000c598\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" Oct 30 04:26:20.778022 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.777971 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2fl9\" (UniqueName: \"kubernetes.io/projected/fcae537d-1ab4-4ab4-b68d-08bd1000c598-kube-api-access-n2fl9\") pod \"payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt\" (UID: \"fcae537d-1ab4-4ab4-b68d-08bd1000c598\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" Oct 30 04:26:20.778102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.778084 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961d5aff-b1b9-4605-806a-7837bdb324a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r\" (UID: \"961d5aff-b1b9-4605-806a-7837bdb324a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" Oct 30 04:26:20.879785 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.879751 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961d5aff-b1b9-4605-806a-7837bdb324a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r\" (UID: \"961d5aff-b1b9-4605-806a-7837bdb324a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" Oct 30 04:26:20.879981 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.879965 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svftc\" (UniqueName: \"kubernetes.io/projected/961d5aff-b1b9-4605-806a-7837bdb324a7-kube-api-access-svftc\") pod \"payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r\" (UID: \"961d5aff-b1b9-4605-806a-7837bdb324a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" Oct 30 04:26:20.880060 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.880002 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcae537d-1ab4-4ab4-b68d-08bd1000c598-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt\" (UID: \"fcae537d-1ab4-4ab4-b68d-08bd1000c598\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" Oct 30 04:26:20.880173 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.880055 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2fl9\" (UniqueName: \"kubernetes.io/projected/fcae537d-1ab4-4ab4-b68d-08bd1000c598-kube-api-access-n2fl9\") pod \"payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt\" (UID: \"fcae537d-1ab4-4ab4-b68d-08bd1000c598\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" Oct 30 04:26:20.880848 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.880825 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcae537d-1ab4-4ab4-b68d-08bd1000c598-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt\" (UID: \"fcae537d-1ab4-4ab4-b68d-08bd1000c598\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" Oct 30 04:26:20.881028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.881004 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961d5aff-b1b9-4605-806a-7837bdb324a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r\" (UID: \"961d5aff-b1b9-4605-806a-7837bdb324a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" Oct 30 04:26:20.903743 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.903719 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svftc\" (UniqueName: \"kubernetes.io/projected/961d5aff-b1b9-4605-806a-7837bdb324a7-kube-api-access-svftc\") pod \"payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r\" (UID: \"961d5aff-b1b9-4605-806a-7837bdb324a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" Oct 30 04:26:20.903743 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.903743 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2fl9\" (UniqueName: \"kubernetes.io/projected/fcae537d-1ab4-4ab4-b68d-08bd1000c598-kube-api-access-n2fl9\") pod \"payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt\" (UID: \"fcae537d-1ab4-4ab4-b68d-08bd1000c598\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" Oct 30 04:26:20.978696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.978668 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" Oct 30 04:26:20.988942 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:20.988919 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" Oct 30 04:26:21.104572 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:21.104470 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:21.296142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:21.296054 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r"] Oct 30 04:26:21.300048 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:26:21.300017 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961d5aff_b1b9_4605_806a_7837bdb324a7.slice/crio-0b83098c9e9e7b1fd04cf3e437871dfc4eba5bd4f195300613392c23d7f8739a WatchSource:0}: Error finding container 0b83098c9e9e7b1fd04cf3e437871dfc4eba5bd4f195300613392c23d7f8739a: Status 404 returned error can't find the container with id 0b83098c9e9e7b1fd04cf3e437871dfc4eba5bd4f195300613392c23d7f8739a Oct 30 04:26:21.319005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:21.318974 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt"] Oct 30 04:26:21.319236 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:26:21.319210 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcae537d_1ab4_4ab4_b68d_08bd1000c598.slice/crio-728ee03edeccbf50b307819c99985d343763f929de3a414e3a96c7a2fd081cb2 WatchSource:0}: Error finding container 728ee03edeccbf50b307819c99985d343763f929de3a414e3a96c7a2fd081cb2: Status 404 returned error can't find the container with id 728ee03edeccbf50b307819c99985d343763f929de3a414e3a96c7a2fd081cb2 Oct 30 04:26:21.722040 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:21.722010 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:22.222615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.222580 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" event={"ID":"fcae537d-1ab4-4ab4-b68d-08bd1000c598","Type":"ContainerStarted","Data":"41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b"} Oct 30 04:26:22.222758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.222637 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" event={"ID":"fcae537d-1ab4-4ab4-b68d-08bd1000c598","Type":"ContainerStarted","Data":"728ee03edeccbf50b307819c99985d343763f929de3a414e3a96c7a2fd081cb2"} Oct 30 04:26:22.222758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.222671 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" Oct 30 04:26:22.224408 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.224298 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" event={"ID":"961d5aff-b1b9-4605-806a-7837bdb324a7","Type":"ContainerStarted","Data":"77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588"} Oct 30 04:26:22.224408 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.224343 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" event={"ID":"961d5aff-b1b9-4605-806a-7837bdb324a7","Type":"ContainerStarted","Data":"0b83098c9e9e7b1fd04cf3e437871dfc4eba5bd4f195300613392c23d7f8739a"} Oct 30 04:26:22.224408 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.224376 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" Oct 30 04:26:22.225782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.225752 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:26:22.226223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.226200 4793 generic.go:356] "Generic (PLEG): container finished" podID="e5e56157-cea5-4efb-86a3-797f64913b05" containerID="073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6" exitCode=0 Oct 30 04:26:22.226304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.226228 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" event={"ID":"e5e56157-cea5-4efb-86a3-797f64913b05","Type":"ContainerDied","Data":"073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6"} Oct 30 04:26:22.226304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.226270 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" event={"ID":"e5e56157-cea5-4efb-86a3-797f64913b05","Type":"ContainerDied","Data":"0185f951b2a299aaa6ad01a64fa5faaec2c41970410b298e079d528939e50ba7"} Oct 30 04:26:22.226304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.226291 4793 scope.go:122] "RemoveContainer" containerID="073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6" Oct 30 04:26:22.226472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.226457 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:22.227871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.227856 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:26:22.237025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.237004 4793 scope.go:122] "RemoveContainer" containerID="3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7" Oct 30 04:26:22.246772 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.246752 4793 scope.go:122] "RemoveContainer" containerID="073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6" Oct 30 04:26:22.247076 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:26:22.247058 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6\": container with ID starting with 073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6 not found: ID does not exist" containerID="073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6" Oct 30 04:26:22.247146 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.247086 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6"} err="failed to get container status \"073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6\": rpc error: code = NotFound desc = could not find container \"073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6\": container with ID starting with 073e2f18de3d2740e003fda127bae3cd8e763b3f4a7dddc2aa43f845a03fe4b6 not found: ID does not exist" Oct 30 04:26:22.247146 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.247111 4793 scope.go:122] "RemoveContainer" containerID="3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7" Oct 30 04:26:22.247402 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:26:22.247383 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7\": container with ID starting with 3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7 not found: ID does not exist" containerID="3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7" Oct 30 04:26:22.247506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.247405 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7"} err="failed to get container status \"3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7\": rpc error: code = NotFound desc = could not find container \"3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7\": container with ID starting with 3e84e808093852fb9f9b2b1b128ba90e9724a9111d515fc42dab15aa00378aa7 not found: ID does not exist" Oct 30 04:26:22.268521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.268467 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" podStartSLOduration=2.268450876 podStartE2EDuration="2.268450876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:22.263679059 +0000 UTC m=+1705.549553835" watchObservedRunningTime="2026-10-30 04:26:22.268450876 +0000 UTC m=+1705.554325654" Oct 30 04:26:22.310066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.309986 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" podStartSLOduration=2.309969056 podStartE2EDuration="2.309969056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:22.304747656 +0000 UTC m=+1705.590622431" watchObservedRunningTime="2026-10-30 04:26:22.309969056 +0000 UTC m=+1705.595843832" Oct 30 04:26:22.314607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.314584 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-credential-socket\") pod \"e5e56157-cea5-4efb-86a3-797f64913b05\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " Oct 30 04:26:22.314703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.314622 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5e56157-cea5-4efb-86a3-797f64913b05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e56157-cea5-4efb-86a3-797f64913b05-istio-podinfo\") pod \"e5e56157-cea5-4efb-86a3-797f64913b05\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " Oct 30 04:26:22.314703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.314685 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-data\") pod \"e5e56157-cea5-4efb-86a3-797f64913b05\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " Oct 30 04:26:22.314787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.314704 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-socket\") pod \"e5e56157-cea5-4efb-86a3-797f64913b05\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " Oct 30 04:26:22.314787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.314754 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-kube-api-access-fmqsc\" (UniqueName: \"kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-kube-api-access-fmqsc\") pod \"e5e56157-cea5-4efb-86a3-797f64913b05\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " Oct 30 04:26:22.314899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.314821 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-credential-socket" pod "e5e56157-cea5-4efb-86a3-797f64913b05" (UID: "e5e56157-cea5-4efb-86a3-797f64913b05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:26:22.314899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.314882 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-certs\") pod \"e5e56157-cea5-4efb-86a3-797f64913b05\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " Oct 30 04:26:22.314997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.314957 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-data" pod "e5e56157-cea5-4efb-86a3-797f64913b05" (UID: "e5e56157-cea5-4efb-86a3-797f64913b05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:26:22.314997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.314983 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5e56157-cea5-4efb-86a3-797f64913b05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e56157-cea5-4efb-86a3-797f64913b05-istiod-ca-cert\") pod \"e5e56157-cea5-4efb-86a3-797f64913b05\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " Oct 30 04:26:22.315099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.315018 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-socket" pod "e5e56157-cea5-4efb-86a3-797f64913b05" (UID: "e5e56157-cea5-4efb-86a3-797f64913b05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:26:22.315099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.315057 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-envoy\") pod \"e5e56157-cea5-4efb-86a3-797f64913b05\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " Oct 30 04:26:22.315201 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.315124 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-certs" pod "e5e56157-cea5-4efb-86a3-797f64913b05" (UID: "e5e56157-cea5-4efb-86a3-797f64913b05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:26:22.315201 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.315151 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-istio-token\" (UniqueName: \"kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-istio-token\") pod \"e5e56157-cea5-4efb-86a3-797f64913b05\" (UID: \"e5e56157-cea5-4efb-86a3-797f64913b05\") " Oct 30 04:26:22.315357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.315328 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5e56157-cea5-4efb-86a3-797f64913b05-istiod-ca-cert" pod "e5e56157-cea5-4efb-86a3-797f64913b05" (UID: "e5e56157-cea5-4efb-86a3-797f64913b05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:26:22.315986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.315963 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:22.316089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.315992 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:22.316089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.316011 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:22.316089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.316025 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5e56157-cea5-4efb-86a3-797f64913b05-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:22.316089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.316039 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:22.317477 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.317453 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-kube-api-access-fmqsc" pod "e5e56157-cea5-4efb-86a3-797f64913b05" (UID: "e5e56157-cea5-4efb-86a3-797f64913b05"). InnerVolumeSpecName "kube-api-access-fmqsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:26:22.317671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.317645 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-envoy" pod "e5e56157-cea5-4efb-86a3-797f64913b05" (UID: "e5e56157-cea5-4efb-86a3-797f64913b05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:26:22.317768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.317715 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-istio-token" pod "e5e56157-cea5-4efb-86a3-797f64913b05" (UID: "e5e56157-cea5-4efb-86a3-797f64913b05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:26:22.318531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.318506 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5e56157-cea5-4efb-86a3-797f64913b05-istio-podinfo" pod "e5e56157-cea5-4efb-86a3-797f64913b05" (UID: "e5e56157-cea5-4efb-86a3-797f64913b05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:26:22.417950 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.417913 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5e56157-cea5-4efb-86a3-797f64913b05-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:22.417950 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.417953 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:22.418147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.417967 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5e56157-cea5-4efb-86a3-797f64913b05-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:22.418147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.417980 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmqsc\" (UniqueName: \"kubernetes.io/projected/e5e56157-cea5-4efb-86a3-797f64913b05-kube-api-access-fmqsc\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:22.679034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.678995 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:22.909486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:22.909450 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:23.125246 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:23.125133 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:23.235540 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:23.235502 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5" Oct 30 04:26:23.305267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:23.305238 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5"] Oct 30 04:26:23.317338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:23.317307 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-397bda-openshift-default-54899c56f9-mddj5"] Oct 30 04:26:24.250250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.250222 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm"] Oct 30 04:26:24.250731 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.250390 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" containerID="cri-o://78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12" gracePeriod=30 Oct 30 04:26:24.467531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.467461 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:24.560655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.560573 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr"] Oct 30 04:26:24.561216 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.561196 4793 cpu_manager.go:414] "containerMap: removing container" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" Oct 30 04:26:24.561325 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.561218 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" Oct 30 04:26:24.561409 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.561396 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" Oct 30 04:26:24.566509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.566487 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" Oct 30 04:26:24.584764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.584738 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-baa474-dockercfg-d5q4l\"" Oct 30 04:26:24.585687 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.585665 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr"] Oct 30 04:26:24.617514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.617490 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-baa474\"" Oct 30 04:26:24.642371 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.642338 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr\" (UID: \"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" Oct 30 04:26:24.642531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.642405 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhzs4\" (UniqueName: \"kubernetes.io/projected/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-kube-api-access-rhzs4\") pod \"payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr\" (UID: \"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" Oct 30 04:26:24.690774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.690738 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:24.744410 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.744372 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr\" (UID: \"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" Oct 30 04:26:24.744604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.744476 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhzs4\" (UniqueName: \"kubernetes.io/projected/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-kube-api-access-rhzs4\") pod \"payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr\" (UID: \"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" Oct 30 04:26:24.745116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.745098 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr\" (UID: \"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" Oct 30 04:26:24.768025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.768001 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhzs4\" (UniqueName: \"kubernetes.io/projected/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-kube-api-access-rhzs4\") pod \"payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr\" (UID: \"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" Oct 30 04:26:24.887491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.885320 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" Oct 30 04:26:24.892471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.891491 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" path="/var/lib/kubelet/pods/e5e56157-cea5-4efb-86a3-797f64913b05/volumes" Oct 30 04:26:24.908440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:24.906234 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:25.216526 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:26:25.216493 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90e1eb6_e8c9_40f6_9a5f_35c3a97e0990.slice/crio-d0b9dbe0c5861ed31ebd5de681e00cbad524c9d828fd8cc5391fc2bc074a2953 WatchSource:0}: Error finding container d0b9dbe0c5861ed31ebd5de681e00cbad524c9d828fd8cc5391fc2bc074a2953: Status 404 returned error can't find the container with id d0b9dbe0c5861ed31ebd5de681e00cbad524c9d828fd8cc5391fc2bc074a2953 Oct 30 04:26:25.216526 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:25.216514 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr"] Oct 30 04:26:25.252779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:25.252754 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" event={"ID":"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990","Type":"ContainerStarted","Data":"d0b9dbe0c5861ed31ebd5de681e00cbad524c9d828fd8cc5391fc2bc074a2953"} Oct 30 04:26:26.267257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:26.267216 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" event={"ID":"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990","Type":"ContainerStarted","Data":"a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de"} Oct 30 04:26:26.267834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:26.267460 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" Oct 30 04:26:26.309600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:26.309548 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" podStartSLOduration=2.3095347889999998 podStartE2EDuration="2.309534789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:26.307998423 +0000 UTC m=+1709.593873197" watchObservedRunningTime="2026-10-30 04:26:26.309534789 +0000 UTC m=+1709.595409564" Oct 30 04:26:26.743098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:26.743063 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:27.060199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:27.060120 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:26:27.060199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:27.060184 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:26:27.184194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:27.184165 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" Oct 30 04:26:28.190494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:28.190466 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" Oct 30 04:26:28.429355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:28.429317 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:29.372966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.372934 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:29.532400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.532345 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:26:29.726557 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.726527 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-certs\") pod \"0886d884-9043-4114-ba66-829bf6f319b7\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " Oct 30 04:26:29.726837 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.726586 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-credential-socket\") pod \"0886d884-9043-4114-ba66-829bf6f319b7\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " Oct 30 04:26:29.726837 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.726615 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-istio-token\" (UniqueName: \"kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-istio-token\") pod \"0886d884-9043-4114-ba66-829bf6f319b7\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " Oct 30 04:26:29.726837 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.726662 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-kube-api-access-vbj8h\" (UniqueName: \"kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-kube-api-access-vbj8h\") pod \"0886d884-9043-4114-ba66-829bf6f319b7\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " Oct 30 04:26:29.726837 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.726717 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-data\") pod \"0886d884-9043-4114-ba66-829bf6f319b7\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " Oct 30 04:26:29.726837 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.726772 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0886d884-9043-4114-ba66-829bf6f319b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0886d884-9043-4114-ba66-829bf6f319b7-istiod-ca-cert\") pod \"0886d884-9043-4114-ba66-829bf6f319b7\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " Oct 30 04:26:29.726837 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.726836 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-socket\") pod \"0886d884-9043-4114-ba66-829bf6f319b7\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " Oct 30 04:26:29.727179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.726853 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-certs" pod "0886d884-9043-4114-ba66-829bf6f319b7" (UID: "0886d884-9043-4114-ba66-829bf6f319b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:26:29.727179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.726929 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-envoy\") pod \"0886d884-9043-4114-ba66-829bf6f319b7\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " Oct 30 04:26:29.727179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.726969 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-credential-socket" pod "0886d884-9043-4114-ba66-829bf6f319b7" (UID: "0886d884-9043-4114-ba66-829bf6f319b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:26:29.727179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.726977 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0886d884-9043-4114-ba66-829bf6f319b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0886d884-9043-4114-ba66-829bf6f319b7-istio-podinfo\") pod \"0886d884-9043-4114-ba66-829bf6f319b7\" (UID: \"0886d884-9043-4114-ba66-829bf6f319b7\") " Oct 30 04:26:29.727179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.727020 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-data" pod "0886d884-9043-4114-ba66-829bf6f319b7" (UID: "0886d884-9043-4114-ba66-829bf6f319b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:26:29.727179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.727170 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-socket" pod "0886d884-9043-4114-ba66-829bf6f319b7" (UID: "0886d884-9043-4114-ba66-829bf6f319b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:26:29.727547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.727207 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0886d884-9043-4114-ba66-829bf6f319b7-istiod-ca-cert" pod "0886d884-9043-4114-ba66-829bf6f319b7" (UID: "0886d884-9043-4114-ba66-829bf6f319b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:26:29.727724 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.727703 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:29.727796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.727732 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:29.727796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.727748 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0886d884-9043-4114-ba66-829bf6f319b7-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:29.727796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.727762 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:29.727796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.727777 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:29.728995 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.728966 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0886d884-9043-4114-ba66-829bf6f319b7-istio-podinfo" pod "0886d884-9043-4114-ba66-829bf6f319b7" (UID: "0886d884-9043-4114-ba66-829bf6f319b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:26:29.729111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.729048 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-kube-api-access-vbj8h" pod "0886d884-9043-4114-ba66-829bf6f319b7" (UID: "0886d884-9043-4114-ba66-829bf6f319b7"). InnerVolumeSpecName "kube-api-access-vbj8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:26:29.729111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.729103 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-envoy" pod "0886d884-9043-4114-ba66-829bf6f319b7" (UID: "0886d884-9043-4114-ba66-829bf6f319b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:26:29.729187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.729140 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-istio-token" pod "0886d884-9043-4114-ba66-829bf6f319b7" (UID: "0886d884-9043-4114-ba66-829bf6f319b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:26:29.833504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.831529 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:29.833504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.831567 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbj8h\" (UniqueName: \"kubernetes.io/projected/0886d884-9043-4114-ba66-829bf6f319b7-kube-api-access-vbj8h\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:29.833504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.831584 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0886d884-9043-4114-ba66-829bf6f319b7-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:29.833504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:29.831608 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0886d884-9043-4114-ba66-829bf6f319b7-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:30.127969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.127883 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:30.297854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.297822 4793 generic.go:356] "Generic (PLEG): container finished" podID="0886d884-9043-4114-ba66-829bf6f319b7" containerID="78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12" exitCode=0 Oct 30 04:26:30.298034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.297914 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" Oct 30 04:26:30.298034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.297914 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" event={"ID":"0886d884-9043-4114-ba66-829bf6f319b7","Type":"ContainerDied","Data":"78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12"} Oct 30 04:26:30.298117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.298048 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm" event={"ID":"0886d884-9043-4114-ba66-829bf6f319b7","Type":"ContainerDied","Data":"5b23928b7f1a65736c75eddf899ce0b8774cf9528a700751eeaeb027eb737186"} Oct 30 04:26:30.298117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.298072 4793 scope.go:122] "RemoveContainer" containerID="78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12" Oct 30 04:26:30.308776 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.308758 4793 scope.go:122] "RemoveContainer" containerID="e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a" Oct 30 04:26:30.317697 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.317681 4793 scope.go:122] "RemoveContainer" containerID="78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12" Oct 30 04:26:30.317989 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:26:30.317967 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12\": container with ID starting with 78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12 not found: ID does not exist" containerID="78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12" Oct 30 04:26:30.318051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.317997 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12"} err="failed to get container status \"78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12\": rpc error: code = NotFound desc = could not find container \"78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12\": container with ID starting with 78ce80b1e414a17c29a3824f13ac52259d74d7b9818deea2f11841c429a5ee12 not found: ID does not exist" Oct 30 04:26:30.318051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.318016 4793 scope.go:122] "RemoveContainer" containerID="e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a" Oct 30 04:26:30.318261 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:26:30.318247 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a\": container with ID starting with e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a not found: ID does not exist" containerID="e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a" Oct 30 04:26:30.318298 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.318264 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a"} err="failed to get container status \"e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a\": rpc error: code = NotFound desc = could not find container \"e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a\": container with ID starting with e6898a5c67bceac548b9366a40b4d4c9269db1b1d4754473033f1fddf31bf23a not found: ID does not exist" Oct 30 04:26:30.365150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.365128 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm"] Oct 30 04:26:30.378180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.378116 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-djckm"] Oct 30 04:26:30.846898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:30.846865 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0886d884-9043-4114-ba66-829bf6f319b7" path="/var/lib/kubelet/pods/0886d884-9043-4114-ba66-829bf6f319b7/volumes" Oct 30 04:26:31.838307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:31.838267 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:33.254622 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:33.254592 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" Oct 30 04:26:33.267440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:33.263250 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" Oct 30 04:26:33.495472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:33.495174 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:33.742178 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:33.742127 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:33.961668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:33.961628 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:35.394057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:35.394016 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:37.114074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:37.114035 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:37.297756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:37.297714 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" Oct 30 04:26:38.773834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.773794 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b"] Oct 30 04:26:38.787593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.787563 4793 cpu_manager.go:414] "containerMap: removing container" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" Oct 30 04:26:38.787791 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.787774 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" Oct 30 04:26:38.787939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.787925 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" Oct 30 04:26:38.788035 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.788022 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" Oct 30 04:26:38.788157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.788144 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" Oct 30 04:26:38.788244 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.788232 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" Oct 30 04:26:38.788488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.788473 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5e56157-cea5-4efb-86a3-797f64913b05" containerName="istio-proxy" Oct 30 04:26:38.788604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.788593 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" Oct 30 04:26:38.796191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.796165 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b"] Oct 30 04:26:38.796334 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.796299 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:38.812059 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.812035 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-baa474-serving-cert\"" Oct 30 04:26:38.824591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.824569 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 04:26:38.824787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.824694 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gw5bk\"" Oct 30 04:26:38.824917 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.824696 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:26:38.825051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.824847 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:26:38.884473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.869125 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:38.884790 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.884763 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb5sw\" (UniqueName: \"kubernetes.io/projected/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-kube-api-access-jb5sw\") pod \"maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:38.884929 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.884914 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-metrics-certs\") pod \"maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:38.885089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.885072 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:38.963960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.963924 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:38.986522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.986485 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:38.986727 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.986559 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb5sw\" (UniqueName: \"kubernetes.io/projected/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-kube-api-access-jb5sw\") pod \"maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:38.986727 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.986627 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-metrics-certs\") pod \"maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:38.986727 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.986707 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:38.988182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.988156 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:38.990908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.990879 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-metrics-certs\") pod \"maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:38.991013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:38.990939 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:39.029374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:39.022213 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb5sw\" (UniqueName: \"kubernetes.io/projected/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-kube-api-access-jb5sw\") pod \"maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:39.123628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:39.123592 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:39.191868 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:39.191771 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:39.484010 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:39.483977 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b"] Oct 30 04:26:39.492690 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:26:39.491617 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea80f40_4200_4e8c_9cfe_8e030f7a39f3.slice/crio-d58aa7b10090bebf2a9aa7f25cdf6b6554f9f96d1029b0af71542154483eaf1f WatchSource:0}: Error finding container d58aa7b10090bebf2a9aa7f25cdf6b6554f9f96d1029b0af71542154483eaf1f: Status 404 returned error can't find the container with id d58aa7b10090bebf2a9aa7f25cdf6b6554f9f96d1029b0af71542154483eaf1f Oct 30 04:26:39.493531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:39.493509 4793 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:26:40.430628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:40.430587 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" event={"ID":"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3","Type":"ContainerStarted","Data":"d58aa7b10090bebf2a9aa7f25cdf6b6554f9f96d1029b0af71542154483eaf1f"} Oct 30 04:26:40.719456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:40.719396 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:41.220343 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:26:41.220308 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:26:41.440972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:41.440912 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" event={"ID":"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3","Type":"ContainerStarted","Data":"0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a"} Oct 30 04:26:41.441451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:41.441115 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:41.485906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:41.485793 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" podStartSLOduration=1.7196717430000001 podStartE2EDuration="3.485775217s" totalImagesPullingTime="1.766103474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:26:41.484381896 +0000 UTC m=+1724.770256674" watchObservedRunningTime="2026-10-30 04:26:41.485775217 +0000 UTC m=+1724.771649993" Oct 30 04:26:42.026619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:42.026574 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:26:42.026859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:42.026640 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:26:42.412774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:42.412675 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:44.288963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:44.288921 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:44.468538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:44.468503 4793 generic.go:356] "Generic (PLEG): container finished" podID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerID="bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98" exitCode=0 Oct 30 04:26:44.468750 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:44.468589 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" event={"ID":"f62df4ee-f0ca-4788-8723-1619ec96c657","Type":"ContainerDied","Data":"bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98"} Oct 30 04:26:44.469075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:44.469057 4793 scope.go:122] "RemoveContainer" containerID="bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98" Oct 30 04:26:44.470595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:44.470566 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:26:44.470678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:44.470635 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:26:44.470678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:44.470666 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:26:45.491209 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:45.491178 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" event={"ID":"f62df4ee-f0ca-4788-8723-1619ec96c657","Type":"ContainerStarted","Data":"1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707"} Oct 30 04:26:45.784570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:45.781950 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:46.097977 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:46.097894 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:46.463441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:46.462945 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:46.469801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:46.466517 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 04:26:46.469801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:46.466569 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 04:26:47.461438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:47.461317 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 04:26:47.461438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:47.461388 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 04:26:47.475439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:47.470556 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:26:47.788499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:47.788364 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx"] Oct 30 04:26:47.817862 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:47.817830 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm"] Oct 30 04:26:48.088498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:48.079104 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:48.299720 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:48.299690 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:48.474438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:48.471501 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 04:26:48.474438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:48.471569 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 04:26:49.470441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:49.468447 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 04:26:49.470441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:49.468515 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 04:26:49.723407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:49.723315 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:50.460221 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:50.460180 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Oct 30 04:26:50.460445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:50.460246 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Oct 30 04:26:51.325546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:51.325511 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:51.464282 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:51.464248 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:51.464757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:51.464719 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:51.465602 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:51.465576 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:26:52.801461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:52.797135 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" podUID="c5c35ef0-7aa2-4a52-9463-800df5b6cd19" containerName="payload-processing" containerID="cri-o://76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2" gracePeriod=25 Oct 30 04:26:52.822073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:52.818912 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" podUID="cb4fb7a2-740e-4ccf-8255-557aeb555b8d" containerName="payload-pre-processing" containerID="cri-o://e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f" gracePeriod=25 Oct 30 04:26:52.857626 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:52.857595 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:53.074304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.074071 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:53.121766 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.121723 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" Oct 30 04:26:53.173462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.172226 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-plugins-config-volume\") pod \"c5c35ef0-7aa2-4a52-9463-800df5b6cd19\" (UID: \"c5c35ef0-7aa2-4a52-9463-800df5b6cd19\") " Oct 30 04:26:53.173462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.172383 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-kube-api-access-x5sr5\" (UniqueName: \"kubernetes.io/projected/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-kube-api-access-x5sr5\") pod \"c5c35ef0-7aa2-4a52-9463-800df5b6cd19\" (UID: \"c5c35ef0-7aa2-4a52-9463-800df5b6cd19\") " Oct 30 04:26:53.173462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.172597 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-plugins-config-volume" pod "c5c35ef0-7aa2-4a52-9463-800df5b6cd19" (UID: "c5c35ef0-7aa2-4a52-9463-800df5b6cd19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:26:53.173462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.172981 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:53.180358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.180338 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" Oct 30 04:26:53.185549 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.185523 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-kube-api-access-x5sr5" pod "c5c35ef0-7aa2-4a52-9463-800df5b6cd19" (UID: "c5c35ef0-7aa2-4a52-9463-800df5b6cd19"). InnerVolumeSpecName "kube-api-access-x5sr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:26:53.274019 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.273986 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-kube-api-access-w9rjj\" (UniqueName: \"kubernetes.io/projected/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-kube-api-access-w9rjj\") pod \"cb4fb7a2-740e-4ccf-8255-557aeb555b8d\" (UID: \"cb4fb7a2-740e-4ccf-8255-557aeb555b8d\") " Oct 30 04:26:53.274193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.274169 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-plugins-config-volume\") pod \"cb4fb7a2-740e-4ccf-8255-557aeb555b8d\" (UID: \"cb4fb7a2-740e-4ccf-8255-557aeb555b8d\") " Oct 30 04:26:53.274616 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.274587 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-plugins-config-volume" pod "cb4fb7a2-740e-4ccf-8255-557aeb555b8d" (UID: "cb4fb7a2-740e-4ccf-8255-557aeb555b8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:26:53.274748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.274690 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5sr5\" (UniqueName: \"kubernetes.io/projected/c5c35ef0-7aa2-4a52-9463-800df5b6cd19-kube-api-access-x5sr5\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:53.276096 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.276076 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-kube-api-access-w9rjj" pod "cb4fb7a2-740e-4ccf-8255-557aeb555b8d" (UID: "cb4fb7a2-740e-4ccf-8255-557aeb555b8d"). InnerVolumeSpecName "kube-api-access-w9rjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:26:53.378535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.377465 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:53.378535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.377511 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rjj\" (UniqueName: \"kubernetes.io/projected/cb4fb7a2-740e-4ccf-8255-557aeb555b8d-kube-api-access-w9rjj\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:53.404634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.404598 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:53.710616 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.710577 4793 generic.go:356] "Generic (PLEG): container finished" podID="c5c35ef0-7aa2-4a52-9463-800df5b6cd19" containerID="76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2" exitCode=0 Oct 30 04:26:53.710825 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.710625 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" Oct 30 04:26:53.710825 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.710658 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" event={"ID":"c5c35ef0-7aa2-4a52-9463-800df5b6cd19","Type":"ContainerDied","Data":"76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2"} Oct 30 04:26:53.710825 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.710712 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx" event={"ID":"c5c35ef0-7aa2-4a52-9463-800df5b6cd19","Type":"ContainerDied","Data":"c2f2126db93e460ef6ebfe6169c2368b2da697a191b422c14b92bb506d324185"} Oct 30 04:26:53.710825 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.710742 4793 scope.go:122] "RemoveContainer" containerID="76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2" Oct 30 04:26:53.712392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.712366 4793 generic.go:356] "Generic (PLEG): container finished" podID="cb4fb7a2-740e-4ccf-8255-557aeb555b8d" containerID="e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f" exitCode=0 Oct 30 04:26:53.712539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.712429 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" Oct 30 04:26:53.712539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.712458 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" event={"ID":"cb4fb7a2-740e-4ccf-8255-557aeb555b8d","Type":"ContainerDied","Data":"e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f"} Oct 30 04:26:53.712539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.712513 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm" event={"ID":"cb4fb7a2-740e-4ccf-8255-557aeb555b8d","Type":"ContainerDied","Data":"0d07de0ecd413ba771c677ea84c172175c943e1bb7fd1bfe073dbb1b078c62cb"} Oct 30 04:26:53.726113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.726092 4793 scope.go:122] "RemoveContainer" containerID="76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2" Oct 30 04:26:53.730369 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:26:53.726493 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2\": container with ID starting with 76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2 not found: ID does not exist" containerID="76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2" Oct 30 04:26:53.730369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.726535 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2"} err="failed to get container status \"76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2\": rpc error: code = NotFound desc = could not find container \"76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2\": container with ID starting with 76b5d762c9859a8445ce6c9c414d0abfc23daf565bd4036fed7ab414799a09c2 not found: ID does not exist" Oct 30 04:26:53.730369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.726561 4793 scope.go:122] "RemoveContainer" containerID="e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f" Oct 30 04:26:53.738680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.738653 4793 scope.go:122] "RemoveContainer" containerID="e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f" Oct 30 04:26:53.738955 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:26:53.738936 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f\": container with ID starting with e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f not found: ID does not exist" containerID="e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f" Oct 30 04:26:53.739026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.738962 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f"} err="failed to get container status \"e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f\": rpc error: code = NotFound desc = could not find container \"e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f\": container with ID starting with e36c026ac74714a641802311af0ce4f701584faa5ffd7f664de83aaab696693f not found: ID does not exist" Oct 30 04:26:53.780478 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.780228 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx"] Oct 30 04:26:53.785896 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.785867 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6dfcfcdd98-rcvcx"] Oct 30 04:26:53.821535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.821505 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm"] Oct 30 04:26:53.825577 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:53.825553 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf4238-5df7887b75-s7svm"] Oct 30 04:26:54.223050 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:54.223018 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:54.250535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:54.250509 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt"] Oct 30 04:26:54.284651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:54.284614 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r"] Oct 30 04:26:54.494962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:54.494862 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:54.716373 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:54.716336 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:54.848931 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:54.848852 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5c35ef0-7aa2-4a52-9463-800df5b6cd19" path="/var/lib/kubelet/pods/c5c35ef0-7aa2-4a52-9463-800df5b6cd19/volumes" Oct 30 04:26:54.849359 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:54.849339 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb4fb7a2-740e-4ccf-8255-557aeb555b8d" path="/var/lib/kubelet/pods/cb4fb7a2-740e-4ccf-8255-557aeb555b8d/volumes" Oct 30 04:26:54.949816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:54.945462 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:56.516985 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:56.516946 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:58.206441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:58.206400 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:26:59.251533 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.251462 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" podUID="fcae537d-1ab4-4ab4-b68d-08bd1000c598" containerName="payload-processing" containerID="cri-o://41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b" gracePeriod=25 Oct 30 04:26:59.285156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.285122 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" podUID="961d5aff-b1b9-4605-806a-7837bdb324a7" containerName="payload-pre-processing" containerID="cri-o://77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588" gracePeriod=25 Oct 30 04:26:59.586582 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.586547 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" Oct 30 04:26:59.618390 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.618362 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" Oct 30 04:26:59.693439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.693076 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961d5aff-b1b9-4605-806a-7837bdb324a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961d5aff-b1b9-4605-806a-7837bdb324a7-plugins-config-volume\") pod \"961d5aff-b1b9-4605-806a-7837bdb324a7\" (UID: \"961d5aff-b1b9-4605-806a-7837bdb324a7\") " Oct 30 04:26:59.693439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.693185 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961d5aff-b1b9-4605-806a-7837bdb324a7-kube-api-access-svftc\" (UniqueName: \"kubernetes.io/projected/961d5aff-b1b9-4605-806a-7837bdb324a7-kube-api-access-svftc\") pod \"961d5aff-b1b9-4605-806a-7837bdb324a7\" (UID: \"961d5aff-b1b9-4605-806a-7837bdb324a7\") " Oct 30 04:26:59.693439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.693405 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcae537d-1ab4-4ab4-b68d-08bd1000c598-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcae537d-1ab4-4ab4-b68d-08bd1000c598-plugins-config-volume\") pod \"fcae537d-1ab4-4ab4-b68d-08bd1000c598\" (UID: \"fcae537d-1ab4-4ab4-b68d-08bd1000c598\") " Oct 30 04:26:59.693744 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.693488 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcae537d-1ab4-4ab4-b68d-08bd1000c598-kube-api-access-n2fl9\" (UniqueName: \"kubernetes.io/projected/fcae537d-1ab4-4ab4-b68d-08bd1000c598-kube-api-access-n2fl9\") pod \"fcae537d-1ab4-4ab4-b68d-08bd1000c598\" (UID: \"fcae537d-1ab4-4ab4-b68d-08bd1000c598\") " Oct 30 04:26:59.700069 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.694630 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961d5aff-b1b9-4605-806a-7837bdb324a7-plugins-config-volume" pod "961d5aff-b1b9-4605-806a-7837bdb324a7" (UID: "961d5aff-b1b9-4605-806a-7837bdb324a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:26:59.700069 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.694678 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcae537d-1ab4-4ab4-b68d-08bd1000c598-plugins-config-volume" pod "fcae537d-1ab4-4ab4-b68d-08bd1000c598" (UID: "fcae537d-1ab4-4ab4-b68d-08bd1000c598"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:26:59.700069 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.695396 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcae537d-1ab4-4ab4-b68d-08bd1000c598-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:59.700069 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.695412 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/961d5aff-b1b9-4605-806a-7837bdb324a7-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:59.700069 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.696711 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcae537d-1ab4-4ab4-b68d-08bd1000c598-kube-api-access-n2fl9" pod "fcae537d-1ab4-4ab4-b68d-08bd1000c598" (UID: "fcae537d-1ab4-4ab4-b68d-08bd1000c598"). InnerVolumeSpecName "kube-api-access-n2fl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:26:59.700846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.700805 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961d5aff-b1b9-4605-806a-7837bdb324a7-kube-api-access-svftc" pod "961d5aff-b1b9-4605-806a-7837bdb324a7" (UID: "961d5aff-b1b9-4605-806a-7837bdb324a7"). InnerVolumeSpecName "kube-api-access-svftc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:26:59.759597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.759501 4793 generic.go:356] "Generic (PLEG): container finished" podID="fcae537d-1ab4-4ab4-b68d-08bd1000c598" containerID="41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b" exitCode=0 Oct 30 04:26:59.759597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.759557 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" Oct 30 04:26:59.759597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.759588 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" event={"ID":"fcae537d-1ab4-4ab4-b68d-08bd1000c598","Type":"ContainerDied","Data":"41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b"} Oct 30 04:26:59.759811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.759641 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt" event={"ID":"fcae537d-1ab4-4ab4-b68d-08bd1000c598","Type":"ContainerDied","Data":"728ee03edeccbf50b307819c99985d343763f929de3a414e3a96c7a2fd081cb2"} Oct 30 04:26:59.759811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.759673 4793 scope.go:122] "RemoveContainer" containerID="41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b" Oct 30 04:26:59.762504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.762314 4793 generic.go:356] "Generic (PLEG): container finished" podID="961d5aff-b1b9-4605-806a-7837bdb324a7" containerID="77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588" exitCode=0 Oct 30 04:26:59.762504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.762389 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" Oct 30 04:26:59.762504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.762428 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" event={"ID":"961d5aff-b1b9-4605-806a-7837bdb324a7","Type":"ContainerDied","Data":"77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588"} Oct 30 04:26:59.762504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.762469 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r" event={"ID":"961d5aff-b1b9-4605-806a-7837bdb324a7","Type":"ContainerDied","Data":"0b83098c9e9e7b1fd04cf3e437871dfc4eba5bd4f195300613392c23d7f8739a"} Oct 30 04:26:59.773315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.773298 4793 scope.go:122] "RemoveContainer" containerID="41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b" Oct 30 04:26:59.773794 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:26:59.773753 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b\": container with ID starting with 41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b not found: ID does not exist" containerID="41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b" Oct 30 04:26:59.773888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.773786 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b"} err="failed to get container status \"41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b\": rpc error: code = NotFound desc = could not find container \"41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b\": container with ID starting with 41ee2b9968e02df8086da98999fb7dbdb112afacbcaa360b8a2e0cb811a2dd6b not found: ID does not exist" Oct 30 04:26:59.773888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.773810 4793 scope.go:122] "RemoveContainer" containerID="77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588" Oct 30 04:26:59.787089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.787068 4793 scope.go:122] "RemoveContainer" containerID="77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588" Oct 30 04:26:59.787479 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:26:59.787403 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588\": container with ID starting with 77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588 not found: ID does not exist" containerID="77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588" Oct 30 04:26:59.787479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.787452 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588"} err="failed to get container status \"77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588\": rpc error: code = NotFound desc = could not find container \"77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588\": container with ID starting with 77ca093572b6f22ea125a3291cc0fc78f337c1ebb92c4cad584a2d0525cab588 not found: ID does not exist" Oct 30 04:26:59.797293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.797246 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2fl9\" (UniqueName: \"kubernetes.io/projected/fcae537d-1ab4-4ab4-b68d-08bd1000c598-kube-api-access-n2fl9\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:59.797293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.797270 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svftc\" (UniqueName: \"kubernetes.io/projected/961d5aff-b1b9-4605-806a-7837bdb324a7-kube-api-access-svftc\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:26:59.818459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.818393 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r"] Oct 30 04:26:59.822083 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.822059 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6dcea5-7fdfbf74ff-nv59r"] Oct 30 04:26:59.865433 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.865394 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt"] Oct 30 04:26:59.865584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.865458 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-74c54fbdd8-79nmt"] Oct 30 04:26:59.992570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:26:59.992531 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:00.857715 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:00.857674 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961d5aff-b1b9-4605-806a-7837bdb324a7" path="/var/lib/kubelet/pods/961d5aff-b1b9-4605-806a-7837bdb324a7/volumes" Oct 30 04:27:00.858304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:00.858238 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcae537d-1ab4-4ab4-b68d-08bd1000c598" path="/var/lib/kubelet/pods/fcae537d-1ab4-4ab4-b68d-08bd1000c598/volumes" Oct 30 04:27:01.696358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.696315 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq"] Oct 30 04:27:01.697595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.697566 4793 cpu_manager.go:414] "containerMap: removing container" podUID="961d5aff-b1b9-4605-806a-7837bdb324a7" containerName="payload-pre-processing" Oct 30 04:27:01.697595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.697593 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961d5aff-b1b9-4605-806a-7837bdb324a7" containerName="payload-pre-processing" Oct 30 04:27:01.697595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.697606 4793 cpu_manager.go:414] "containerMap: removing container" podUID="fcae537d-1ab4-4ab4-b68d-08bd1000c598" containerName="payload-processing" Oct 30 04:27:01.698203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.697614 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcae537d-1ab4-4ab4-b68d-08bd1000c598" containerName="payload-processing" Oct 30 04:27:01.698203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.697688 4793 cpu_manager.go:414] "containerMap: removing container" podUID="cb4fb7a2-740e-4ccf-8255-557aeb555b8d" containerName="payload-pre-processing" Oct 30 04:27:01.698203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.697696 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb4fb7a2-740e-4ccf-8255-557aeb555b8d" containerName="payload-pre-processing" Oct 30 04:27:01.698203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.697707 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c5c35ef0-7aa2-4a52-9463-800df5b6cd19" containerName="payload-processing" Oct 30 04:27:01.698203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.697714 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5c35ef0-7aa2-4a52-9463-800df5b6cd19" containerName="payload-processing" Oct 30 04:27:01.698203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.698145 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="0886d884-9043-4114-ba66-829bf6f319b7" containerName="istio-proxy" Oct 30 04:27:01.698203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.698193 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb4fb7a2-740e-4ccf-8255-557aeb555b8d" containerName="payload-pre-processing" Oct 30 04:27:01.698203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.698205 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5c35ef0-7aa2-4a52-9463-800df5b6cd19" containerName="payload-processing" Oct 30 04:27:01.698673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.698223 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="961d5aff-b1b9-4605-806a-7837bdb324a7" containerName="payload-pre-processing" Oct 30 04:27:01.698673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.698234 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcae537d-1ab4-4ab4-b68d-08bd1000c598" containerName="payload-processing" Oct 30 04:27:01.706700 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.706674 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq"] Oct 30 04:27:01.706809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.706712 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" Oct 30 04:27:01.722548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.722521 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf4238\"" Oct 30 04:27:01.722657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.722619 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf4238-dockercfg-hfmp6\"" Oct 30 04:27:01.803082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.803053 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:01.825193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.825163 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4rp\" (UniqueName: \"kubernetes.io/projected/8a2c9185-072d-4655-806d-7434d227252c-kube-api-access-tn4rp\") pod \"payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq\" (UID: \"8a2c9185-072d-4655-806d-7434d227252c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" Oct 30 04:27:01.825329 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.825307 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2c9185-072d-4655-806d-7434d227252c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq\" (UID: \"8a2c9185-072d-4655-806d-7434d227252c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" Oct 30 04:27:01.926911 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.926876 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4rp\" (UniqueName: \"kubernetes.io/projected/8a2c9185-072d-4655-806d-7434d227252c-kube-api-access-tn4rp\") pod \"payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq\" (UID: \"8a2c9185-072d-4655-806d-7434d227252c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" Oct 30 04:27:01.927361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.926936 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2c9185-072d-4655-806d-7434d227252c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq\" (UID: \"8a2c9185-072d-4655-806d-7434d227252c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" Oct 30 04:27:01.927781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.927755 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2c9185-072d-4655-806d-7434d227252c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq\" (UID: \"8a2c9185-072d-4655-806d-7434d227252c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" Oct 30 04:27:01.951306 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:01.951248 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4rp\" (UniqueName: \"kubernetes.io/projected/8a2c9185-072d-4655-806d-7434d227252c-kube-api-access-tn4rp\") pod \"payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq\" (UID: \"8a2c9185-072d-4655-806d-7434d227252c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" Oct 30 04:27:02.021929 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:02.021894 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" Oct 30 04:27:02.336115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:02.336084 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq"] Oct 30 04:27:02.336514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:02.336411 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:02.795653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:02.795142 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" event={"ID":"8a2c9185-072d-4655-806d-7434d227252c","Type":"ContainerStarted","Data":"517169f9f75739526331882bdcbf19739701746df9f3631ffa8fa3c66c4cdc1e"} Oct 30 04:27:02.797303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:02.797268 4793 generic.go:356] "Generic (PLEG): container finished" podID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerID="32ece55d2c07dfd9eced0b507e04ad174c042f5d548e7a7a7b8acd293be1ba5a" exitCode=0 Oct 30 04:27:02.797486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:02.797459 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" event={"ID":"5dc09d10-3490-43e5-9d1f-f52fed67d037","Type":"ContainerDied","Data":"32ece55d2c07dfd9eced0b507e04ad174c042f5d548e7a7a7b8acd293be1ba5a"} Oct 30 04:27:02.797916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:02.797895 4793 scope.go:122] "RemoveContainer" containerID="32ece55d2c07dfd9eced0b507e04ad174c042f5d548e7a7a7b8acd293be1ba5a" Oct 30 04:27:02.799670 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:02.799625 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:27:02.799776 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:02.799699 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:27:02.799776 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:02.799736 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:27:03.193233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:03.193189 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr"] Oct 30 04:27:03.445702 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:03.445532 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:03.747002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:03.746912 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:03.808308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:03.808265 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" event={"ID":"5dc09d10-3490-43e5-9d1f-f52fed67d037","Type":"ContainerStarted","Data":"ebc1654ad9e3bf3fd29e6cc29229738b62a125f1e04d021227259041aa16fc84"} Oct 30 04:27:04.001197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:04.001105 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:04.217436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:04.216780 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:04.716730 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:04.716698 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:27:04.718143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:04.718114 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 04:27:04.718261 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:04.718187 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 04:27:04.817330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:04.817297 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" event={"ID":"8a2c9185-072d-4655-806d-7434d227252c","Type":"ContainerStarted","Data":"448e2b84969e8ecdabd264162eda26314346ca1b3f17cda262153900aa0c9de1"} Oct 30 04:27:04.817546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:04.817379 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" Oct 30 04:27:04.886669 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:04.886558 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" podStartSLOduration=1.5908216099999999 podStartE2EDuration="3.886538163s" totalImagesPullingTime="2.295716553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:04.868667493 +0000 UTC m=+1748.154542274" watchObservedRunningTime="2026-10-30 04:27:04.886538163 +0000 UTC m=+1748.172412941" Oct 30 04:27:05.716218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:05.716183 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 04:27:05.716699 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:05.716246 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 04:27:06.019311 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:06.019227 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:06.334266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:06.334165 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:06.568443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:06.566628 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:06.716625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:06.716592 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 04:27:06.717014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:06.716645 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 04:27:06.793062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:06.793022 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:07.226523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:07.226492 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:27:07.716454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:07.716406 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 04:27:07.716647 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:07.716475 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 04:27:07.751884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:07.751851 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:07.977938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:07.977856 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:08.193687 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.193637 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" podUID="c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990" containerName="payload-processing" containerID="cri-o://a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de" gracePeriod=25 Oct 30 04:27:08.526150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.526123 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" Oct 30 04:27:08.689501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.678881 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-plugins-config-volume\") pod \"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990\" (UID: \"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990\") " Oct 30 04:27:08.689501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.678974 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-kube-api-access-rhzs4\" (UniqueName: \"kubernetes.io/projected/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-kube-api-access-rhzs4\") pod \"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990\" (UID: \"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990\") " Oct 30 04:27:08.690319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.690269 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-plugins-config-volume" pod "c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990" (UID: "c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:27:08.693679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.693509 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-kube-api-access-rhzs4" pod "c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990" (UID: "c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990"). InnerVolumeSpecName "kube-api-access-rhzs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:27:08.717152 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.717131 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 30 04:27:08.717482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.717173 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 30 04:27:08.782434 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.782279 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:08.782434 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.782309 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhzs4\" (UniqueName: \"kubernetes.io/projected/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990-kube-api-access-rhzs4\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:08.912733 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.912697 4793 generic.go:356] "Generic (PLEG): container finished" podID="c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990" containerID="a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de" exitCode=0 Oct 30 04:27:08.912733 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.912740 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" Oct 30 04:27:08.912969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.912778 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" event={"ID":"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990","Type":"ContainerDied","Data":"a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de"} Oct 30 04:27:08.912969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.912833 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr" event={"ID":"c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990","Type":"ContainerDied","Data":"d0b9dbe0c5861ed31ebd5de681e00cbad524c9d828fd8cc5391fc2bc074a2953"} Oct 30 04:27:08.912969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.912857 4793 scope.go:122] "RemoveContainer" containerID="a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de" Oct 30 04:27:08.925368 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.925346 4793 scope.go:122] "RemoveContainer" containerID="a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de" Oct 30 04:27:08.925695 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:27:08.925671 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de\": container with ID starting with a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de not found: ID does not exist" containerID="a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de" Oct 30 04:27:08.925759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.925704 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de"} err="failed to get container status \"a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de\": rpc error: code = NotFound desc = could not find container \"a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de\": container with ID starting with a672d634f60a9e92ca13d0264722826e8c49a14aed5c5eabb1150909f2e197de not found: ID does not exist" Oct 30 04:27:08.968572 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.968483 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr"] Oct 30 04:27:08.971671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:08.971642 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-baa474-65c6d8f6f4-v4cbr"] Oct 30 04:27:09.575497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:09.575386 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:09.720765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:09.720734 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:27:09.721657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:09.721634 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:27:10.830443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:10.823657 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:10.906441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:10.902642 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990" path="/var/lib/kubelet/pods/c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990/volumes" Oct 30 04:27:11.037698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.037656 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:11.219285 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:27:11.219244 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:27:11.286236 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.286206 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r"] Oct 30 04:27:11.286889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.286869 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990" containerName="payload-processing" Oct 30 04:27:11.286889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.286890 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990" containerName="payload-processing" Oct 30 04:27:11.287076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.287007 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90e1eb6-e8c9-40f6-9a5f-35c3a97e0990" containerName="payload-processing" Oct 30 04:27:11.290362 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.290341 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" Oct 30 04:27:11.308981 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.308959 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r"] Oct 30 04:27:11.317598 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.317572 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6dcea5\"" Oct 30 04:27:11.317856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.317823 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6dcea5-dockercfg-vk549\"" Oct 30 04:27:11.416377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.416342 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/304cae42-e227-4708-a6c4-20c2b7cc9768-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r\" (UID: \"304cae42-e227-4708-a6c4-20c2b7cc9768\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" Oct 30 04:27:11.416562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.416403 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjm87\" (UniqueName: \"kubernetes.io/projected/304cae42-e227-4708-a6c4-20c2b7cc9768-kube-api-access-wjm87\") pod \"payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r\" (UID: \"304cae42-e227-4708-a6c4-20c2b7cc9768\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" Oct 30 04:27:11.519502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.517676 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/304cae42-e227-4708-a6c4-20c2b7cc9768-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r\" (UID: \"304cae42-e227-4708-a6c4-20c2b7cc9768\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" Oct 30 04:27:11.519502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.517757 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjm87\" (UniqueName: \"kubernetes.io/projected/304cae42-e227-4708-a6c4-20c2b7cc9768-kube-api-access-wjm87\") pod \"payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r\" (UID: \"304cae42-e227-4708-a6c4-20c2b7cc9768\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" Oct 30 04:27:11.519502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.518573 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/304cae42-e227-4708-a6c4-20c2b7cc9768-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r\" (UID: \"304cae42-e227-4708-a6c4-20c2b7cc9768\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" Oct 30 04:27:11.542479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.542448 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjm87\" (UniqueName: \"kubernetes.io/projected/304cae42-e227-4708-a6c4-20c2b7cc9768-kube-api-access-wjm87\") pod \"payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r\" (UID: \"304cae42-e227-4708-a6c4-20c2b7cc9768\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" Oct 30 04:27:11.604730 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.604700 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" Oct 30 04:27:11.908913 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:27:11.908881 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod304cae42_e227_4708_a6c4_20c2b7cc9768.slice/crio-60f2887d6e98cd7319381298298ca489d0dec0ae699aea202375a26c670dba65 WatchSource:0}: Error finding container 60f2887d6e98cd7319381298298ca489d0dec0ae699aea202375a26c670dba65: Status 404 returned error can't find the container with id 60f2887d6e98cd7319381298298ca489d0dec0ae699aea202375a26c670dba65 Oct 30 04:27:11.909667 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.909636 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r"] Oct 30 04:27:11.946543 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.946490 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" event={"ID":"304cae42-e227-4708-a6c4-20c2b7cc9768","Type":"ContainerStarted","Data":"60f2887d6e98cd7319381298298ca489d0dec0ae699aea202375a26c670dba65"} Oct 30 04:27:11.986200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:11.986167 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:12.956036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:12.955997 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" event={"ID":"304cae42-e227-4708-a6c4-20c2b7cc9768","Type":"ContainerStarted","Data":"33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b"} Oct 30 04:27:12.956574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:12.956261 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" Oct 30 04:27:13.000062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:13.000013 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" podStartSLOduration=1.999999995 podStartE2EDuration="1.999999995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:12.997551714 +0000 UTC m=+1756.283426490" watchObservedRunningTime="2026-10-30 04:27:12.999999995 +0000 UTC m=+1756.285874769" Oct 30 04:27:13.951602 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:13.951438 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4"] Oct 30 04:27:13.966511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:13.966473 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" podUID="389a8074-a44c-42c7-8696-775753993e07" containerName="istio-proxy" containerID="cri-o://d0c7e8c06b501788b64d1125aa680c2bb8493dfcf2652bb58ec734945fcf4792" gracePeriod=30 Oct 30 04:27:14.029690 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:14.029651 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:15.821756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:15.821716 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:15.827720 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:15.827698 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" Oct 30 04:27:16.040665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:16.040630 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:17.091226 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:17.091195 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:17.329208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:17.329174 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:18.217757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:18.217719 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:19.005000 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.004970 4793 generic.go:356] "Generic (PLEG): container finished" podID="389a8074-a44c-42c7-8696-775753993e07" containerID="d0c7e8c06b501788b64d1125aa680c2bb8493dfcf2652bb58ec734945fcf4792" exitCode=0 Oct 30 04:27:19.005102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.005042 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" event={"ID":"389a8074-a44c-42c7-8696-775753993e07","Type":"ContainerDied","Data":"d0c7e8c06b501788b64d1125aa680c2bb8493dfcf2652bb58ec734945fcf4792"} Oct 30 04:27:19.631998 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.631975 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:27:19.723760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.723727 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-data\") pod \"389a8074-a44c-42c7-8696-775753993e07\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " Oct 30 04:27:19.723947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.723805 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-socket\") pod \"389a8074-a44c-42c7-8696-775753993e07\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " Oct 30 04:27:19.723947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.723843 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/389a8074-a44c-42c7-8696-775753993e07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389a8074-a44c-42c7-8696-775753993e07-istio-podinfo\") pod \"389a8074-a44c-42c7-8696-775753993e07\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " Oct 30 04:27:19.723947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.723886 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389a8074-a44c-42c7-8696-775753993e07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389a8074-a44c-42c7-8696-775753993e07-istiod-ca-cert\") pod \"389a8074-a44c-42c7-8696-775753993e07\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " Oct 30 04:27:19.723947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.723914 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-envoy\") pod \"389a8074-a44c-42c7-8696-775753993e07\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " Oct 30 04:27:19.724180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.723957 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-kube-api-access-kt8rn\" (UniqueName: \"kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-kube-api-access-kt8rn\") pod \"389a8074-a44c-42c7-8696-775753993e07\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " Oct 30 04:27:19.724180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.723998 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-credential-socket\") pod \"389a8074-a44c-42c7-8696-775753993e07\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " Oct 30 04:27:19.724180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.724020 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-data" pod "389a8074-a44c-42c7-8696-775753993e07" (UID: "389a8074-a44c-42c7-8696-775753993e07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:27:19.724180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.724051 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-istio-token\" (UniqueName: \"kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-istio-token\") pod \"389a8074-a44c-42c7-8696-775753993e07\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " Oct 30 04:27:19.724180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.724117 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-socket" pod "389a8074-a44c-42c7-8696-775753993e07" (UID: "389a8074-a44c-42c7-8696-775753993e07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:27:19.724180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.724143 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-certs\") pod \"389a8074-a44c-42c7-8696-775753993e07\" (UID: \"389a8074-a44c-42c7-8696-775753993e07\") " Oct 30 04:27:19.725167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.724678 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389a8074-a44c-42c7-8696-775753993e07-istiod-ca-cert" pod "389a8074-a44c-42c7-8696-775753993e07" (UID: "389a8074-a44c-42c7-8696-775753993e07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:27:19.725167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.724794 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:19.725167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.724878 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-certs" pod "389a8074-a44c-42c7-8696-775753993e07" (UID: "389a8074-a44c-42c7-8696-775753993e07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:27:19.725167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.724909 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/389a8074-a44c-42c7-8696-775753993e07-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:19.725167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.724924 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:19.725759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.725732 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-credential-socket" pod "389a8074-a44c-42c7-8696-775753993e07" (UID: "389a8074-a44c-42c7-8696-775753993e07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:27:19.726839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.726809 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/389a8074-a44c-42c7-8696-775753993e07-istio-podinfo" pod "389a8074-a44c-42c7-8696-775753993e07" (UID: "389a8074-a44c-42c7-8696-775753993e07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:27:19.727193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.727166 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-envoy" pod "389a8074-a44c-42c7-8696-775753993e07" (UID: "389a8074-a44c-42c7-8696-775753993e07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:27:19.727330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.727306 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-kube-api-access-kt8rn" pod "389a8074-a44c-42c7-8696-775753993e07" (UID: "389a8074-a44c-42c7-8696-775753993e07"). InnerVolumeSpecName "kube-api-access-kt8rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:27:19.728300 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.728279 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-istio-token" pod "389a8074-a44c-42c7-8696-775753993e07" (UID: "389a8074-a44c-42c7-8696-775753993e07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:27:19.826186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.826144 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/389a8074-a44c-42c7-8696-775753993e07-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:19.826186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.826186 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:19.826488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.826201 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt8rn\" (UniqueName: \"kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-kube-api-access-kt8rn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:19.826488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.826217 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:19.826488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.826231 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/389a8074-a44c-42c7-8696-775753993e07-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:19.826488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.826246 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/389a8074-a44c-42c7-8696-775753993e07-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:19.874661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:19.874629 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:20.015166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.015076 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" Oct 30 04:27:20.015166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.015089 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4" event={"ID":"389a8074-a44c-42c7-8696-775753993e07","Type":"ContainerDied","Data":"84814657d80e5384c91ab13e54d63545e23415392cdf477905cb86ed7e2eec0e"} Oct 30 04:27:20.015166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.015160 4793 scope.go:122] "RemoveContainer" containerID="d0c7e8c06b501788b64d1125aa680c2bb8493dfcf2652bb58ec734945fcf4792" Oct 30 04:27:20.081268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.080990 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4"] Oct 30 04:27:20.086659 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.086627 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-f2xf4"] Oct 30 04:27:20.158076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.158034 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:20.401767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.400989 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw"] Oct 30 04:27:20.408854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.402468 4793 cpu_manager.go:414] "containerMap: removing container" podUID="389a8074-a44c-42c7-8696-775753993e07" containerName="istio-proxy" Oct 30 04:27:20.408854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.402513 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389a8074-a44c-42c7-8696-775753993e07" containerName="istio-proxy" Oct 30 04:27:20.408854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.402700 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="389a8074-a44c-42c7-8696-775753993e07" containerName="istio-proxy" Oct 30 04:27:20.410486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.410464 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.432264 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.432240 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istiod-ca-cert\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.432364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.432287 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shh49\" (UniqueName: \"kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-kube-api-access-shh49\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.432364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.432352 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-certs\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.432457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.432403 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-token\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.432517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.432472 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-socket\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.432569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.432554 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-data\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.432613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.432600 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-envoy\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.432646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.432624 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-credential-socket\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.432756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.432735 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-podinfo\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.434370 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.434354 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ca7a81-openshift-default-dockercfg-k5dvf\"" Oct 30 04:27:20.449598 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.449578 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw"] Oct 30 04:27:20.546442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.546360 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-token\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.581446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.581269 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-socket\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.581446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.581338 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-data\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.581446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.581377 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-envoy\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.581446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.581406 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-credential-socket\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.581792 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.581504 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-podinfo\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.581792 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.581594 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istiod-ca-cert\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.581792 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.581669 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shh49\" (UniqueName: \"kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-kube-api-access-shh49\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.581792 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.581719 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-certs\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.588444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.582272 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-certs\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.588444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.582574 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-socket\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.588444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.582818 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-data\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.588444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.586305 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-podinfo\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.588444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.586577 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-credential-socket\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.588444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.587175 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istiod-ca-cert\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.624440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.622066 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-token\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.648562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.636153 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:20.648562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.643650 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-envoy\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.648562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.648103 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shh49\" (UniqueName: \"kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-kube-api-access-shh49\") pod \"e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.731871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.731831 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:20.848317 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.847974 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389a8074-a44c-42c7-8696-775753993e07" path="/var/lib/kubelet/pods/389a8074-a44c-42c7-8696-775753993e07/volumes" Oct 30 04:27:20.859220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:20.859171 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:21.069928 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:21.069899 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw"] Oct 30 04:27:21.070504 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:27:21.070476 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0f9d7d_88e3_4dd1_a41b_0ced1bcc7b7f.slice/crio-7da15ccdaadbf9929fac3f20e5aed5451a774fb2192f5b40dcdb3733381b4547 WatchSource:0}: Error finding container 7da15ccdaadbf9929fac3f20e5aed5451a774fb2192f5b40dcdb3733381b4547: Status 404 returned error can't find the container with id 7da15ccdaadbf9929fac3f20e5aed5451a774fb2192f5b40dcdb3733381b4547 Oct 30 04:27:21.073528 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:21.073498 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:27:21.073603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:21.073560 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:27:21.073603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:21.073589 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:27:21.906471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:21.906406 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:22.035679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:22.035645 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" event={"ID":"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f","Type":"ContainerStarted","Data":"0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9"} Oct 30 04:27:22.035862 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:22.035711 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" event={"ID":"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f","Type":"ContainerStarted","Data":"7da15ccdaadbf9929fac3f20e5aed5451a774fb2192f5b40dcdb3733381b4547"} Oct 30 04:27:22.168494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:22.164854 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" podStartSLOduration=2.164834353 podStartE2EDuration="2.164834353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:22.096368927 +0000 UTC m=+1765.382243702" watchObservedRunningTime="2026-10-30 04:27:22.164834353 +0000 UTC m=+1765.450709129" Oct 30 04:27:22.528967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:22.528935 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:22.732875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:22.732845 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:22.737916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:22.737896 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:23.052165 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:23.052133 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:23.053135 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:23.053117 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:27:23.968851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:23.968817 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" Oct 30 04:27:24.290411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:24.290324 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:24.604062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:24.603977 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:24.821010 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:24.820971 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:25.149725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:25.149696 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:25.977745 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:25.977702 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:26.217308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.217278 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:26.367044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.366962 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq"] Oct 30 04:27:26.373080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.373053 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" Oct 30 04:27:26.393276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.393250 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq"] Oct 30 04:27:26.400727 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.400694 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ca7a81\"" Oct 30 04:27:26.420574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.420551 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ca7a81-dockercfg-twv8k\"" Oct 30 04:27:26.452498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.452471 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq"] Oct 30 04:27:26.457888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.457856 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" Oct 30 04:27:26.459764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.459715 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:26.518442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.514817 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq"] Oct 30 04:27:26.525875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.525011 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8btd\" (UniqueName: \"kubernetes.io/projected/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-kube-api-access-l8btd\") pod \"payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq\" (UID: \"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" Oct 30 04:27:26.525875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.525073 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4917caa4-90ee-4220-952c-69ddcd973fdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq\" (UID: \"4917caa4-90ee-4220-952c-69ddcd973fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" Oct 30 04:27:26.525875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.525133 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r5rc\" (UniqueName: \"kubernetes.io/projected/4917caa4-90ee-4220-952c-69ddcd973fdc-kube-api-access-8r5rc\") pod \"payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq\" (UID: \"4917caa4-90ee-4220-952c-69ddcd973fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" Oct 30 04:27:26.525875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.525293 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq\" (UID: \"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" Oct 30 04:27:26.634482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.627492 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq\" (UID: \"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" Oct 30 04:27:26.634482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.627638 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8btd\" (UniqueName: \"kubernetes.io/projected/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-kube-api-access-l8btd\") pod \"payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq\" (UID: \"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" Oct 30 04:27:26.634482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.627679 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4917caa4-90ee-4220-952c-69ddcd973fdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq\" (UID: \"4917caa4-90ee-4220-952c-69ddcd973fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" Oct 30 04:27:26.634482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.627727 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r5rc\" (UniqueName: \"kubernetes.io/projected/4917caa4-90ee-4220-952c-69ddcd973fdc-kube-api-access-8r5rc\") pod \"payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq\" (UID: \"4917caa4-90ee-4220-952c-69ddcd973fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" Oct 30 04:27:26.634482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.628728 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq\" (UID: \"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" Oct 30 04:27:26.641014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.636952 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4917caa4-90ee-4220-952c-69ddcd973fdc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq\" (UID: \"4917caa4-90ee-4220-952c-69ddcd973fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" Oct 30 04:27:26.654352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.654327 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r5rc\" (UniqueName: \"kubernetes.io/projected/4917caa4-90ee-4220-952c-69ddcd973fdc-kube-api-access-8r5rc\") pod \"payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq\" (UID: \"4917caa4-90ee-4220-952c-69ddcd973fdc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" Oct 30 04:27:26.654787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.654769 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8btd\" (UniqueName: \"kubernetes.io/projected/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-kube-api-access-l8btd\") pod \"payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq\" (UID: \"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" Oct 30 04:27:26.685553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.685512 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:26.689309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.689282 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" Oct 30 04:27:26.940481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:26.940451 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" Oct 30 04:27:27.011477 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:27:27.010924 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4917caa4_90ee_4220_952c_69ddcd973fdc.slice/crio-35d7af7068b2d0bc207b09de6feeac6ae0c6531f84442a45f7b6c73f4afc88d0 WatchSource:0}: Error finding container 35d7af7068b2d0bc207b09de6feeac6ae0c6531f84442a45f7b6c73f4afc88d0: Status 404 returned error can't find the container with id 35d7af7068b2d0bc207b09de6feeac6ae0c6531f84442a45f7b6c73f4afc88d0 Oct 30 04:27:27.015254 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:27.015224 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq"] Oct 30 04:27:27.083769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:27.083733 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" event={"ID":"4917caa4-90ee-4220-952c-69ddcd973fdc","Type":"ContainerStarted","Data":"35d7af7068b2d0bc207b09de6feeac6ae0c6531f84442a45f7b6c73f4afc88d0"} Oct 30 04:27:27.244144 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:27:27.244116 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca56ef9_9b23_4d9d_85b0_f63a79f0f8f6.slice/crio-f47d7f1fb196270091939f53c118565ea6b753072786f4d9624c5f6a78302772 WatchSource:0}: Error finding container f47d7f1fb196270091939f53c118565ea6b753072786f4d9624c5f6a78302772: Status 404 returned error can't find the container with id f47d7f1fb196270091939f53c118565ea6b753072786f4d9624c5f6a78302772 Oct 30 04:27:27.244599 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:27.244579 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq"] Oct 30 04:27:28.038811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:28.038770 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:28.092161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:28.092129 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" event={"ID":"4917caa4-90ee-4220-952c-69ddcd973fdc","Type":"ContainerStarted","Data":"af86a13a47ec0d5e7ef72869b79fae7862e4c4a95dfeccf3b53ac882c5fdb016"} Oct 30 04:27:28.092363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:28.092236 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" Oct 30 04:27:28.093799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:28.093778 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" event={"ID":"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6","Type":"ContainerStarted","Data":"06012c97a46e77bf5bf9c0677275ca61837f2f1367f34b52eebb987d97ce3445"} Oct 30 04:27:28.093891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:28.093814 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" event={"ID":"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6","Type":"ContainerStarted","Data":"f47d7f1fb196270091939f53c118565ea6b753072786f4d9624c5f6a78302772"} Oct 30 04:27:28.093891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:28.093834 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" Oct 30 04:27:28.136456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:28.136393 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" podStartSLOduration=2.136382021 podStartE2EDuration="2.136382021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:28.133044698 +0000 UTC m=+1771.418919473" watchObservedRunningTime="2026-10-30 04:27:28.136382021 +0000 UTC m=+1771.422256795" Oct 30 04:27:28.175172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:28.175120 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" podStartSLOduration=2.175109382 podStartE2EDuration="2.175109382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:28.173701026 +0000 UTC m=+1771.459575815" watchObservedRunningTime="2026-10-30 04:27:28.175109382 +0000 UTC m=+1771.460984156" Oct 30 04:27:30.060103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:30.060066 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:30.288248 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:30.288207 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:30.520349 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:30.520305 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:31.838301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:31.838264 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:31.887560 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:31.887522 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq"] Oct 30 04:27:31.892434 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:31.892398 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" Oct 30 04:27:31.910585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:31.910557 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq"] Oct 30 04:27:31.969882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:31.969852 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b1a417-dockercfg-6mhx4\"" Oct 30 04:27:31.970217 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:31.970201 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b1a417\"" Oct 30 04:27:32.006668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:32.006627 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60aae71a-d64a-497b-b48f-f4767178434a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq\" (UID: \"60aae71a-d64a-497b-b48f-f4767178434a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" Oct 30 04:27:32.007205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:32.007166 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdj2\" (UniqueName: \"kubernetes.io/projected/60aae71a-d64a-497b-b48f-f4767178434a-kube-api-access-lbdj2\") pod \"payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq\" (UID: \"60aae71a-d64a-497b-b48f-f4767178434a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" Oct 30 04:27:32.081521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:32.078765 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:32.109093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:32.108999 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60aae71a-d64a-497b-b48f-f4767178434a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq\" (UID: \"60aae71a-d64a-497b-b48f-f4767178434a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" Oct 30 04:27:32.109281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:32.109250 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdj2\" (UniqueName: \"kubernetes.io/projected/60aae71a-d64a-497b-b48f-f4767178434a-kube-api-access-lbdj2\") pod \"payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq\" (UID: \"60aae71a-d64a-497b-b48f-f4767178434a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" Oct 30 04:27:32.109841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:32.109819 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60aae71a-d64a-497b-b48f-f4767178434a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq\" (UID: \"60aae71a-d64a-497b-b48f-f4767178434a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" Oct 30 04:27:32.133093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:32.133069 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdj2\" (UniqueName: \"kubernetes.io/projected/60aae71a-d64a-497b-b48f-f4767178434a-kube-api-access-lbdj2\") pod \"payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq\" (UID: \"60aae71a-d64a-497b-b48f-f4767178434a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" Oct 30 04:27:32.216843 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:32.216806 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" Oct 30 04:27:32.306900 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:32.306812 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:32.531643 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:32.531604 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:32.531810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:32.531788 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq"] Oct 30 04:27:33.022906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:33.022871 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:33.131386 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:33.131352 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" event={"ID":"60aae71a-d64a-497b-b48f-f4767178434a","Type":"ContainerStarted","Data":"c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43"} Oct 30 04:27:33.131615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:33.131404 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" event={"ID":"60aae71a-d64a-497b-b48f-f4767178434a","Type":"ContainerStarted","Data":"8e43184221b1baef4195a6cd0fda7809cb32596424f121f2dee38a4acca316c7"} Oct 30 04:27:33.131615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:33.131471 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" Oct 30 04:27:33.176315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:33.176258 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" podStartSLOduration=2.176222836 podStartE2EDuration="2.176222836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:33.174840965 +0000 UTC m=+1776.460715740" watchObservedRunningTime="2026-10-30 04:27:33.176222836 +0000 UTC m=+1776.462097613" Oct 30 04:27:33.302175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:33.302076 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:33.537677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:33.537643 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t"] Oct 30 04:27:33.537935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:33.537900 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerName="istio-proxy" containerID="cri-o://a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb" gracePeriod=30 Oct 30 04:27:33.721449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:33.719448 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:34.144509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:34.144427 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:35.467226 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.467195 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:35.797969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.797889 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w"] Oct 30 04:27:35.802538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.802514 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.818138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.818112 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-763bde-openshift-default-dockercfg-h48ph\"" Oct 30 04:27:35.839277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.839241 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w"] Oct 30 04:27:35.861270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.861235 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-socket\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.861467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.861374 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdnq7\" (UniqueName: \"kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-kube-api-access-qdnq7\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.861556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.861528 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-token\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.861688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.861666 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-certs\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.861754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.861706 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-envoy\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.861754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.861736 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-podinfo\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.861871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.861846 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-data\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.861932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.861877 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-credential-socket\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.861932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.861906 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istiod-ca-cert\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.962772 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.962739 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-socket\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.962957 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.962903 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdnq7\" (UniqueName: \"kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-kube-api-access-qdnq7\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.963032 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.962997 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-token\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.963128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.963110 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-certs\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.963199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.963146 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-envoy\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.963199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.963172 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-podinfo\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.963315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.963236 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-data\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.963315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.963252 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-socket\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.963315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.963268 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-credential-socket\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.963315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.963299 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istiod-ca-cert\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.963581 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.963517 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-certs\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.963702 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.963681 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-credential-socket\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.963774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.963709 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:35.963909 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.963878 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-data\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.964282 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.964259 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istiod-ca-cert\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.971816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.965512 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-envoy\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.971816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.965908 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-podinfo\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.986867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.986840 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-token\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:35.987973 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:35.987948 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdnq7\" (UniqueName: \"kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-kube-api-access-qdnq7\") pod \"e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:36.136605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:36.136521 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:36.181509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:36.181474 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:36.462711 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:36.462674 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w"] Oct 30 04:27:36.463041 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:27:36.463013 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc4700a_fc65_4c54_ab00_66df2dd352c5.slice/crio-264771e9fa015111b3edbd08b705442a342ccdac628415a4bce657b9372b255c WatchSource:0}: Error finding container 264771e9fa015111b3edbd08b705442a342ccdac628415a4bce657b9372b255c: Status 404 returned error can't find the container with id 264771e9fa015111b3edbd08b705442a342ccdac628415a4bce657b9372b255c Oct 30 04:27:36.467459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:36.467396 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:27:36.467901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:36.467514 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:27:36.467901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:36.467558 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:27:36.601280 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:36.601243 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:37.162122 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:37.162089 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" event={"ID":"7fc4700a-fc65-4c54-ab00-66df2dd352c5","Type":"ContainerStarted","Data":"dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f"} Oct 30 04:27:37.162290 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:37.162146 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" event={"ID":"7fc4700a-fc65-4c54-ab00-66df2dd352c5","Type":"ContainerStarted","Data":"264771e9fa015111b3edbd08b705442a342ccdac628415a4bce657b9372b255c"} Oct 30 04:27:37.235108 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:37.235053 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" podStartSLOduration=2.235039708 podStartE2EDuration="2.235039708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:37.22053969 +0000 UTC m=+1780.506414466" watchObservedRunningTime="2026-10-30 04:27:37.235039708 +0000 UTC m=+1780.520914530" Oct 30 04:27:38.136965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.136926 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:38.143288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.143264 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:38.169970 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.169949 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:38.170893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.170872 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:27:38.309272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.309234 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:38.818997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.818936 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:27:38.905012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.904979 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-envoy\") pod \"0a10528f-fdbe-4c27-8b64-67adff5df547\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " Oct 30 04:27:38.905208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905109 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-socket\") pod \"0a10528f-fdbe-4c27-8b64-67adff5df547\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " Oct 30 04:27:38.905208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905153 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-token\" (UniqueName: \"kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-token\") pod \"0a10528f-fdbe-4c27-8b64-67adff5df547\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " Oct 30 04:27:38.905208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905177 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-credential-socket\") pod \"0a10528f-fdbe-4c27-8b64-67adff5df547\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " Oct 30 04:27:38.905389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905283 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a10528f-fdbe-4c27-8b64-67adff5df547-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a10528f-fdbe-4c27-8b64-67adff5df547-istiod-ca-cert\") pod \"0a10528f-fdbe-4c27-8b64-67adff5df547\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " Oct 30 04:27:38.905389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905326 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-kube-api-access-dvrjq\" (UniqueName: \"kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-kube-api-access-dvrjq\") pod \"0a10528f-fdbe-4c27-8b64-67adff5df547\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " Oct 30 04:27:38.905389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905364 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-data\") pod \"0a10528f-fdbe-4c27-8b64-67adff5df547\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " Oct 30 04:27:38.905583 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905439 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-podinfo\") pod \"0a10528f-fdbe-4c27-8b64-67adff5df547\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " Oct 30 04:27:38.905583 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905445 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-socket" pod "0a10528f-fdbe-4c27-8b64-67adff5df547" (UID: "0a10528f-fdbe-4c27-8b64-67adff5df547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:27:38.905583 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905471 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-certs\") pod \"0a10528f-fdbe-4c27-8b64-67adff5df547\" (UID: \"0a10528f-fdbe-4c27-8b64-67adff5df547\") " Oct 30 04:27:38.905738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905711 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-credential-socket" pod "0a10528f-fdbe-4c27-8b64-67adff5df547" (UID: "0a10528f-fdbe-4c27-8b64-67adff5df547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:27:38.905797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905744 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-certs" pod "0a10528f-fdbe-4c27-8b64-67adff5df547" (UID: "0a10528f-fdbe-4c27-8b64-67adff5df547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:27:38.905853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905789 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a10528f-fdbe-4c27-8b64-67adff5df547-istiod-ca-cert" pod "0a10528f-fdbe-4c27-8b64-67adff5df547" (UID: "0a10528f-fdbe-4c27-8b64-67adff5df547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:27:38.905972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.905951 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-data" pod "0a10528f-fdbe-4c27-8b64-67adff5df547" (UID: "0a10528f-fdbe-4c27-8b64-67adff5df547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:27:38.906293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.906268 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:38.906402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.906297 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:38.906402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.906315 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a10528f-fdbe-4c27-8b64-67adff5df547-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:38.906402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.906329 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:38.906402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.906341 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:38.907793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.907763 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-envoy" pod "0a10528f-fdbe-4c27-8b64-67adff5df547" (UID: "0a10528f-fdbe-4c27-8b64-67adff5df547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:27:38.907924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.907899 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-podinfo" pod "0a10528f-fdbe-4c27-8b64-67adff5df547" (UID: "0a10528f-fdbe-4c27-8b64-67adff5df547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:27:38.908342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.908321 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-kube-api-access-dvrjq" pod "0a10528f-fdbe-4c27-8b64-67adff5df547" (UID: "0a10528f-fdbe-4c27-8b64-67adff5df547"). InnerVolumeSpecName "kube-api-access-dvrjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:27:38.908611 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:38.908586 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-token" pod "0a10528f-fdbe-4c27-8b64-67adff5df547" (UID: "0a10528f-fdbe-4c27-8b64-67adff5df547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:27:39.007951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.007915 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvrjq\" (UniqueName: \"kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-kube-api-access-dvrjq\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:39.007951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.007945 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:39.008129 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.007958 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:39.008129 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.007970 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a10528f-fdbe-4c27-8b64-67adff5df547-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:27:39.103782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.103703 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" Oct 30 04:27:39.103782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.103777 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" Oct 30 04:27:39.179741 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.179694 4793 generic.go:356] "Generic (PLEG): container finished" podID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerID="a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb" exitCode=0 Oct 30 04:27:39.180611 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.180592 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" Oct 30 04:27:39.184555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.184508 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" event={"ID":"0a10528f-fdbe-4c27-8b64-67adff5df547","Type":"ContainerDied","Data":"a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb"} Oct 30 04:27:39.184555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.184553 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t" event={"ID":"0a10528f-fdbe-4c27-8b64-67adff5df547","Type":"ContainerDied","Data":"2014ac4b10893e7b1f85a643a0291112caf9b9f468c223eb9acae288b91ded34"} Oct 30 04:27:39.184750 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.184603 4793 scope.go:122] "RemoveContainer" containerID="a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb" Oct 30 04:27:39.203157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.203138 4793 scope.go:122] "RemoveContainer" containerID="c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7" Oct 30 04:27:39.214075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.214057 4793 scope.go:122] "RemoveContainer" containerID="a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb" Oct 30 04:27:39.214332 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:27:39.214311 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb\": container with ID starting with a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb not found: ID does not exist" containerID="a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb" Oct 30 04:27:39.214403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.214343 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb"} err="failed to get container status \"a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb\": rpc error: code = NotFound desc = could not find container \"a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb\": container with ID starting with a9918f0d1dd39924de5d29c8b9493b28a332c501fa2d1a4fad5fae1b28ac91fb not found: ID does not exist" Oct 30 04:27:39.214403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.214361 4793 scope.go:122] "RemoveContainer" containerID="c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7" Oct 30 04:27:39.214689 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:27:39.214662 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7\": container with ID starting with c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7 not found: ID does not exist" containerID="c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7" Oct 30 04:27:39.214765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.214698 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7"} err="failed to get container status \"c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7\": rpc error: code = NotFound desc = could not find container \"c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7\": container with ID starting with c4b9fd1c35c56d5a050a25dcccb03cd150b80a52d597fde7f559e55f99aa45d7 not found: ID does not exist" Oct 30 04:27:39.245081 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.245048 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t"] Oct 30 04:27:39.258775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.258754 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1a0d3427-openshift-default-697689cb57-28z5t"] Oct 30 04:27:39.736714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.736681 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:39.958909 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:39.958876 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:40.738620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:40.738571 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:40.846684 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:40.846643 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" path="/var/lib/kubelet/pods/0a10528f-fdbe-4c27-8b64-67adff5df547/volumes" Oct 30 04:27:41.221375 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:27:41.221337 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:27:41.532638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:41.532552 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:41.777442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:41.775869 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:42.367751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.367717 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm"] Oct 30 04:27:42.368283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.368270 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerName="istio-proxy" Oct 30 04:27:42.368332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.368284 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerName="istio-proxy" Oct 30 04:27:42.368397 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.368389 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerName="istio-proxy" Oct 30 04:27:42.374715 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.374693 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:42.407565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.407525 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm"] Oct 30 04:27:42.436653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.436626 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ca7a81-serving-cert\"" Oct 30 04:27:42.453401 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.453374 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:42.453557 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.453522 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-metrics-certs\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:42.453725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.453699 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw8hz\" (UniqueName: \"kubernetes.io/projected/df5168c8-44f4-4ef6-84de-1a9437aca7a2-kube-api-access-dw8hz\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:42.453854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.453757 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5168c8-44f4-4ef6-84de-1a9437aca7a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:42.560963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.560927 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw8hz\" (UniqueName: \"kubernetes.io/projected/df5168c8-44f4-4ef6-84de-1a9437aca7a2-kube-api-access-dw8hz\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:42.561164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.560993 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5168c8-44f4-4ef6-84de-1a9437aca7a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:42.561164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.561120 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:42.561273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.561210 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-metrics-certs\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:42.561430 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:27:42.561386 4793 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-serving-cert: secret "maas-api-e2e-ipp-b-ca7a81-serving-cert" not found Oct 30 04:27:42.561552 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:27:42.561525 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-maas-api-tls podName:df5168c8-44f4-4ef6-84de-1a9437aca7a2 nodeName:}" failed. No retries permitted until 2026-10-30 04:27:43.061503478 +0000 UTC m=+1786.347378234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-maas-api-tls") pod "maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" (UID: "df5168c8-44f4-4ef6-84de-1a9437aca7a2") : secret "maas-api-e2e-ipp-b-ca7a81-serving-cert" not found Oct 30 04:27:42.561836 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.561808 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5168c8-44f4-4ef6-84de-1a9437aca7a2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:42.564707 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.564682 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-metrics-certs\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:42.570216 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.570179 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:42.585749 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.585713 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw8hz\" (UniqueName: \"kubernetes.io/projected/df5168c8-44f4-4ef6-84de-1a9437aca7a2-kube-api-access-dw8hz\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:42.789257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:42.789219 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:43.066273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:43.066190 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:43.068787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:43.068758 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:43.289804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:43.289767 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:43.601928 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:27:43.601845 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5168c8_44f4_4ef6_84de_1a9437aca7a2.slice/crio-4e0012725177114b703cae9cdb80585e725548bac2d19379c7271cfd773a2713 WatchSource:0}: Error finding container 4e0012725177114b703cae9cdb80585e725548bac2d19379c7271cfd773a2713: Status 404 returned error can't find the container with id 4e0012725177114b703cae9cdb80585e725548bac2d19379c7271cfd773a2713 Oct 30 04:27:43.602478 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:43.602456 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm"] Oct 30 04:27:44.141591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:44.141565 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" Oct 30 04:27:44.232205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:44.232113 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" event={"ID":"df5168c8-44f4-4ef6-84de-1a9437aca7a2","Type":"ContainerStarted","Data":"0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1"} Oct 30 04:27:44.232505 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:44.232478 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" event={"ID":"df5168c8-44f4-4ef6-84de-1a9437aca7a2","Type":"ContainerStarted","Data":"4e0012725177114b703cae9cdb80585e725548bac2d19379c7271cfd773a2713"} Oct 30 04:27:44.233635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:44.233614 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:44.255965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:44.255933 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:44.289565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:44.289515 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" podStartSLOduration=1.917664096 podStartE2EDuration="2.289502191s" totalImagesPullingTime="371.838095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:44.287764531 +0000 UTC m=+1787.573639328" watchObservedRunningTime="2026-10-30 04:27:44.289502191 +0000 UTC m=+1787.575376965" Oct 30 04:27:45.912589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:45.912548 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:47.609765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:47.609724 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:49.725444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.720456 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7"] Oct 30 04:27:49.725444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.721549 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerName="istio-proxy" Oct 30 04:27:49.725444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.721569 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerName="istio-proxy" Oct 30 04:27:49.725444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.721751 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a10528f-fdbe-4c27-8b64-67adff5df547" containerName="istio-proxy" Oct 30 04:27:49.726533 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.726506 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:49.741438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.740433 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7"] Oct 30 04:27:49.743467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.742775 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b1a417-serving-cert\"" Oct 30 04:27:49.847460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.835314 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c50c74d-2900-4872-b08b-882118c38c1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:49.847460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.835397 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:49.848214 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.847850 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-metrics-certs\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:49.848214 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.847974 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcr88\" (UniqueName: \"kubernetes.io/projected/3c50c74d-2900-4872-b08b-882118c38c1d-kube-api-access-hcr88\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:49.931452 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.912063 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:49.958262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.956150 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:49.958262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.956227 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-metrics-certs\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:49.958262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.956342 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcr88\" (UniqueName: \"kubernetes.io/projected/3c50c74d-2900-4872-b08b-882118c38c1d-kube-api-access-hcr88\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:49.958262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.956536 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c50c74d-2900-4872-b08b-882118c38c1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:49.958262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:49.957468 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c50c74d-2900-4872-b08b-882118c38c1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:49.958262 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:27:49.957583 4793 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-serving-cert: secret "maas-api-e2e-ipp-b-b1a417-serving-cert" not found Oct 30 04:27:49.958262 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:27:49.957641 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-maas-api-tls podName:3c50c74d-2900-4872-b08b-882118c38c1d nodeName:}" failed. No retries permitted until 2026-10-30 04:27:50.457621171 +0000 UTC m=+1793.743495939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-maas-api-tls") pod "maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" (UID: "3c50c74d-2900-4872-b08b-882118c38c1d") : secret "maas-api-e2e-ipp-b-b1a417-serving-cert" not found Oct 30 04:27:50.021494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:50.012745 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-metrics-certs\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:50.097441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:50.091047 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcr88\" (UniqueName: \"kubernetes.io/projected/3c50c74d-2900-4872-b08b-882118c38c1d-kube-api-access-hcr88\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:50.244336 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:50.244297 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:50.492442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:50.492176 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:50.529441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:50.528993 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:50.534001 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:50.533969 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:50.546443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:50.542382 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:51.310444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:51.299651 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:27:51.359153 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:51.355829 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7"] Oct 30 04:27:51.905994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:51.905665 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:52.359990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:52.359954 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" event={"ID":"3c50c74d-2900-4872-b08b-882118c38c1d","Type":"ContainerStarted","Data":"47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583"} Oct 30 04:27:52.360401 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:52.360017 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" event={"ID":"3c50c74d-2900-4872-b08b-882118c38c1d","Type":"ContainerStarted","Data":"034b8376719f3f4a2dff4162f5e75112413d7022158ebcdc3d57089307aaa46b"} Oct 30 04:27:52.360401 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:52.360097 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:52.404850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:52.404802 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" podStartSLOduration=2.950320079 podStartE2EDuration="3.404789285s" totalImagesPullingTime="454.469206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:27:52.402160648 +0000 UTC m=+1795.688035420" watchObservedRunningTime="2026-10-30 04:27:52.404789285 +0000 UTC m=+1795.690664104" Oct 30 04:27:53.481940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:53.481904 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:53.724658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:53.724625 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:55.648109 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:55.648072 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:57.836886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:57.836799 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:27:58.372244 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:58.372212 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:27:59.536788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:27:59.536752 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:01.749178 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:01.749137 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:01.965679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:01.965641 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:03.579138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:03.579097 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:05.313721 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:05.313686 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:07.018868 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:07.018821 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:08.803481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:08.803447 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:10.352878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:10.352845 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq"] Oct 30 04:28:10.394128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:10.394089 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq"] Oct 30 04:28:10.495037 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:10.495006 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:10.725474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:10.725437 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:10.949468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:10.949431 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:11.221513 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:28:11.221481 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:28:12.799485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:12.799363 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:14.564090 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:14.564050 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:15.353908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.353870 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" podUID="3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6" containerName="payload-processing" containerID="cri-o://06012c97a46e77bf5bf9c0677275ca61837f2f1367f34b52eebb987d97ce3445" gracePeriod=25 Oct 30 04:28:15.394525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.394483 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" podUID="4917caa4-90ee-4220-952c-69ddcd973fdc" containerName="payload-pre-processing" containerID="cri-o://af86a13a47ec0d5e7ef72869b79fae7862e4c4a95dfeccf3b53ac882c5fdb016" gracePeriod=25 Oct 30 04:28:15.559802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.559551 4793 generic.go:356] "Generic (PLEG): container finished" podID="4917caa4-90ee-4220-952c-69ddcd973fdc" containerID="af86a13a47ec0d5e7ef72869b79fae7862e4c4a95dfeccf3b53ac882c5fdb016" exitCode=0 Oct 30 04:28:15.559802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.559706 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" event={"ID":"4917caa4-90ee-4220-952c-69ddcd973fdc","Type":"ContainerDied","Data":"af86a13a47ec0d5e7ef72869b79fae7862e4c4a95dfeccf3b53ac882c5fdb016"} Oct 30 04:28:15.563148 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.563055 4793 generic.go:356] "Generic (PLEG): container finished" podID="3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6" containerID="06012c97a46e77bf5bf9c0677275ca61837f2f1367f34b52eebb987d97ce3445" exitCode=0 Oct 30 04:28:15.563148 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.563102 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" event={"ID":"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6","Type":"ContainerDied","Data":"06012c97a46e77bf5bf9c0677275ca61837f2f1367f34b52eebb987d97ce3445"} Oct 30 04:28:15.689247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.688895 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" Oct 30 04:28:15.767449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.767390 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" Oct 30 04:28:15.800768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.800736 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-kube-api-access-l8btd\" (UniqueName: \"kubernetes.io/projected/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-kube-api-access-l8btd\") pod \"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6\" (UID: \"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6\") " Oct 30 04:28:15.800923 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.800796 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4917caa4-90ee-4220-952c-69ddcd973fdc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4917caa4-90ee-4220-952c-69ddcd973fdc-plugins-config-volume\") pod \"4917caa4-90ee-4220-952c-69ddcd973fdc\" (UID: \"4917caa4-90ee-4220-952c-69ddcd973fdc\") " Oct 30 04:28:15.800988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.800916 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-plugins-config-volume\") pod \"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6\" (UID: \"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6\") " Oct 30 04:28:15.801089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.801067 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4917caa4-90ee-4220-952c-69ddcd973fdc-kube-api-access-8r5rc\" (UniqueName: \"kubernetes.io/projected/4917caa4-90ee-4220-952c-69ddcd973fdc-kube-api-access-8r5rc\") pod \"4917caa4-90ee-4220-952c-69ddcd973fdc\" (UID: \"4917caa4-90ee-4220-952c-69ddcd973fdc\") " Oct 30 04:28:15.801233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.801211 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4917caa4-90ee-4220-952c-69ddcd973fdc-plugins-config-volume" pod "4917caa4-90ee-4220-952c-69ddcd973fdc" (UID: "4917caa4-90ee-4220-952c-69ddcd973fdc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:28:15.801324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.801297 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-plugins-config-volume" pod "3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6" (UID: "3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:28:15.801655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.801635 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4917caa4-90ee-4220-952c-69ddcd973fdc-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:28:15.801758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.801656 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:28:15.803249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.803223 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4917caa4-90ee-4220-952c-69ddcd973fdc-kube-api-access-8r5rc" pod "4917caa4-90ee-4220-952c-69ddcd973fdc" (UID: "4917caa4-90ee-4220-952c-69ddcd973fdc"). InnerVolumeSpecName "kube-api-access-8r5rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:28:15.803362 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.803346 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-kube-api-access-l8btd" pod "3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6" (UID: "3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6"). InnerVolumeSpecName "kube-api-access-l8btd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:28:15.903016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.902932 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8btd\" (UniqueName: \"kubernetes.io/projected/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6-kube-api-access-l8btd\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:28:15.903016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.902962 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r5rc\" (UniqueName: \"kubernetes.io/projected/4917caa4-90ee-4220-952c-69ddcd973fdc-kube-api-access-8r5rc\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:28:15.922015 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:15.921977 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq"] Oct 30 04:28:16.137888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.137841 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:16.355675 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.355559 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:16.576667 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.576570 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" event={"ID":"3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6","Type":"ContainerDied","Data":"f47d7f1fb196270091939f53c118565ea6b753072786f4d9624c5f6a78302772"} Oct 30 04:28:16.576969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.576949 4793 scope.go:122] "RemoveContainer" containerID="06012c97a46e77bf5bf9c0677275ca61837f2f1367f34b52eebb987d97ce3445" Oct 30 04:28:16.577223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.577207 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq" Oct 30 04:28:16.590648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.585754 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" event={"ID":"4917caa4-90ee-4220-952c-69ddcd973fdc","Type":"ContainerDied","Data":"35d7af7068b2d0bc207b09de6feeac6ae0c6531f84442a45f7b6c73f4afc88d0"} Oct 30 04:28:16.590857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.585793 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq" Oct 30 04:28:16.599779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.599750 4793 generic.go:356] "Generic (PLEG): container finished" podID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerID="0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9" exitCode=0 Oct 30 04:28:16.599988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.599947 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" event={"ID":"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f","Type":"ContainerDied","Data":"0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9"} Oct 30 04:28:16.601586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.601565 4793 scope.go:122] "RemoveContainer" containerID="0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9" Oct 30 04:28:16.612571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.612534 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:28:16.612799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.612774 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:28:16.612988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.612965 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:28:16.622352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.622330 4793 scope.go:122] "RemoveContainer" containerID="af86a13a47ec0d5e7ef72869b79fae7862e4c4a95dfeccf3b53ac882c5fdb016" Oct 30 04:28:16.720300 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.720279 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq"] Oct 30 04:28:16.723700 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.723676 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-5cd9997c64-nnbhq"] Oct 30 04:28:16.759048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.759018 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq"] Oct 30 04:28:16.770078 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.770049 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ca7a81-74bc86f77f-gn2cq"] Oct 30 04:28:16.847539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.847461 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6" path="/var/lib/kubelet/pods/3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6/volumes" Oct 30 04:28:16.847854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:16.847839 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4917caa4-90ee-4220-952c-69ddcd973fdc" path="/var/lib/kubelet/pods/4917caa4-90ee-4220-952c-69ddcd973fdc/volumes" Oct 30 04:28:17.158006 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.157906 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:17.546608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.546572 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm"] Oct 30 04:28:17.547772 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.547743 4793 cpu_manager.go:414] "containerMap: removing container" podUID="4917caa4-90ee-4220-952c-69ddcd973fdc" containerName="payload-pre-processing" Oct 30 04:28:17.547936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.547923 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4917caa4-90ee-4220-952c-69ddcd973fdc" containerName="payload-pre-processing" Oct 30 04:28:17.548078 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.548066 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6" containerName="payload-processing" Oct 30 04:28:17.548206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.548146 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6" containerName="payload-processing" Oct 30 04:28:17.548510 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.548489 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ca56ef9-9b23-4d9d-85b0-f63a79f0f8f6" containerName="payload-processing" Oct 30 04:28:17.548665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.548651 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="4917caa4-90ee-4220-952c-69ddcd973fdc" containerName="payload-pre-processing" Oct 30 04:28:17.552712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.552691 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.607443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.589302 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f7a36d-openshift-default-dockercfg-zqsd6\"" Oct 30 04:28:17.614141 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.614082 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm"] Oct 30 04:28:17.630347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.629730 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" event={"ID":"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f","Type":"ContainerStarted","Data":"64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b"} Oct 30 04:28:17.636745 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.636712 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-data\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.636889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.636759 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-socket\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.636889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.636795 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-token\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.636889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.636842 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-envoy\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.637066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.636913 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-credential-socket\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.637066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.636939 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-podinfo\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.637066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.636985 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzk85\" (UniqueName: \"kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-kube-api-access-nzk85\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.637216 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.637150 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istiod-ca-cert\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.637216 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.637192 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-certs\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.697334 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.697298 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:17.740439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.736891 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:28:17.740439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.737024 4793 patch_prober.go:29] interesting pod/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 04:28:17.740439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.737069 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 04:28:17.740439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.738111 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzk85\" (UniqueName: \"kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-kube-api-access-nzk85\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.740439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.738290 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istiod-ca-cert\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.740439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.738329 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-certs\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.740439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.738440 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-data\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.740439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.738473 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-socket\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.740439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.738506 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-token\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.740439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.738558 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-envoy\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.740439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.738650 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-credential-socket\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.740439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.738694 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-podinfo\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.763442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.761971 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istiod-ca-cert\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.763442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.762240 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-certs\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.763442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.762984 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-data\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.763442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.763226 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-socket\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.785446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.780754 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-envoy\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.785446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.781216 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-credential-socket\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.809493 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.807277 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-podinfo\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.831438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.823084 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-token\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.861496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.855121 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzk85\" (UniqueName: \"kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-kube-api-access-nzk85\") pod \"e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:17.916626 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.916593 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:17.925592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:17.925571 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:18.150550 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:18.150469 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:18.291486 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:28:18.291454 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b57c75_ef43_4572_9e9c_2c56102a6d5b.slice/crio-5f2d36a121c0fc6ae69c9fd15b39778608286f688bab7df6324f460f0f0d4728 WatchSource:0}: Error finding container 5f2d36a121c0fc6ae69c9fd15b39778608286f688bab7df6324f460f0f0d4728: Status 404 returned error can't find the container with id 5f2d36a121c0fc6ae69c9fd15b39778608286f688bab7df6324f460f0f0d4728 Oct 30 04:28:18.292579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:18.292557 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm"] Oct 30 04:28:18.294544 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:18.294510 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:28:18.294652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:18.294590 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:28:18.294652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:18.294641 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:28:18.698615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:18.698578 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" event={"ID":"40b57c75-ef43-4572-9e9c-2c56102a6d5b","Type":"ContainerStarted","Data":"e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1"} Oct 30 04:28:18.698800 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:18.698651 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" event={"ID":"40b57c75-ef43-4572-9e9c-2c56102a6d5b","Type":"ContainerStarted","Data":"5f2d36a121c0fc6ae69c9fd15b39778608286f688bab7df6324f460f0f0d4728"} Oct 30 04:28:18.732710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:18.732677 4793 patch_prober.go:29] interesting pod/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 04:28:18.732872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:18.732729 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 04:28:18.746525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:18.746467 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" podStartSLOduration=1.746449022 podStartE2EDuration="1.746449022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:18.744600014 +0000 UTC m=+1822.030474818" watchObservedRunningTime="2026-10-30 04:28:18.746449022 +0000 UTC m=+1822.032323798" Oct 30 04:28:18.927605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:18.926078 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:19.732233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:19.732202 4793 patch_prober.go:29] interesting pod/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 04:28:19.732413 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:19.732261 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 04:28:19.931825 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:19.931788 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:20.499514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:20.499465 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:20.720271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:20.720234 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:20.721880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:20.721490 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:28:20.734709 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:20.734677 4793 patch_prober.go:29] interesting pod/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 04:28:20.734861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:20.734743 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 04:28:20.923497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:20.922913 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" podUID="60aae71a-d64a-497b-b48f-f4767178434a" containerName="payload-pre-processing" containerID="cri-o://c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43" gracePeriod=25 Oct 30 04:28:21.333269 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.332888 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:21.500346 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.500006 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" Oct 30 04:28:21.602932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.601067 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:21.603108 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.602939 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60aae71a-d64a-497b-b48f-f4767178434a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60aae71a-d64a-497b-b48f-f4767178434a-plugins-config-volume\") pod \"60aae71a-d64a-497b-b48f-f4767178434a\" (UID: \"60aae71a-d64a-497b-b48f-f4767178434a\") " Oct 30 04:28:21.603230 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.603162 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60aae71a-d64a-497b-b48f-f4767178434a-kube-api-access-lbdj2\" (UniqueName: \"kubernetes.io/projected/60aae71a-d64a-497b-b48f-f4767178434a-kube-api-access-lbdj2\") pod \"60aae71a-d64a-497b-b48f-f4767178434a\" (UID: \"60aae71a-d64a-497b-b48f-f4767178434a\") " Oct 30 04:28:21.627440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.615941 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60aae71a-d64a-497b-b48f-f4767178434a-plugins-config-volume" pod "60aae71a-d64a-497b-b48f-f4767178434a" (UID: "60aae71a-d64a-497b-b48f-f4767178434a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:28:21.632916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.632783 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60aae71a-d64a-497b-b48f-f4767178434a-kube-api-access-lbdj2" pod "60aae71a-d64a-497b-b48f-f4767178434a" (UID: "60aae71a-d64a-497b-b48f-f4767178434a"). InnerVolumeSpecName "kube-api-access-lbdj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:28:21.713968 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.705242 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbdj2\" (UniqueName: \"kubernetes.io/projected/60aae71a-d64a-497b-b48f-f4767178434a-kube-api-access-lbdj2\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:28:21.713968 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.705276 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60aae71a-d64a-497b-b48f-f4767178434a-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:28:21.728222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.728188 4793 generic.go:356] "Generic (PLEG): container finished" podID="60aae71a-d64a-497b-b48f-f4767178434a" containerID="c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43" exitCode=0 Oct 30 04:28:21.728353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.728228 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" Oct 30 04:28:21.728353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.728273 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" event={"ID":"60aae71a-d64a-497b-b48f-f4767178434a","Type":"ContainerDied","Data":"c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43"} Oct 30 04:28:21.728353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.728328 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq" event={"ID":"60aae71a-d64a-497b-b48f-f4767178434a","Type":"ContainerDied","Data":"8e43184221b1baef4195a6cd0fda7809cb32596424f121f2dee38a4acca316c7"} Oct 30 04:28:21.728353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.728351 4793 scope.go:122] "RemoveContainer" containerID="c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43" Oct 30 04:28:21.732120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.732094 4793 patch_prober.go:29] interesting pod/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 04:28:21.732244 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.732147 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 04:28:21.739990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.739975 4793 scope.go:122] "RemoveContainer" containerID="c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43" Oct 30 04:28:21.740239 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:28:21.740221 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43\": container with ID starting with c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43 not found: ID does not exist" containerID="c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43" Oct 30 04:28:21.740281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.740247 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43"} err="failed to get container status \"c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43\": rpc error: code = NotFound desc = could not find container \"c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43\": container with ID starting with c7b34b391e0f4cc0185cdb4bd5e209d71f7add7f73a9641ebe41225b95fa8f43 not found: ID does not exist" Oct 30 04:28:21.802450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.802371 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq"] Oct 30 04:28:21.810926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:21.810894 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b1a417-7cd6c4966c-vrrxq"] Oct 30 04:28:22.534441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:22.533450 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:22.732432 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:22.732378 4793 patch_prober.go:29] interesting pod/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 04:28:22.732634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:22.732445 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 04:28:22.847737 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:22.847658 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60aae71a-d64a-497b-b48f-f4767178434a" path="/var/lib/kubelet/pods/60aae71a-d64a-497b-b48f-f4767178434a/volumes" Oct 30 04:28:23.052553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:23.052519 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:28:23.191217 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:23.191009 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:23.456320 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:23.456223 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:23.737623 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:23.737552 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:28:23.745523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:23.745502 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:28:24.785981 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:24.785935 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:26.384333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:26.384239 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:28.137752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:28.137715 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:29.334303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:29.334260 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:29.566678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:29.566620 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:29.796510 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:29.796477 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c"] Oct 30 04:28:29.809450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:29.797082 4793 cpu_manager.go:414] "containerMap: removing container" podUID="60aae71a-d64a-497b-b48f-f4767178434a" containerName="payload-pre-processing" Oct 30 04:28:29.809450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:29.797095 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60aae71a-d64a-497b-b48f-f4767178434a" containerName="payload-pre-processing" Oct 30 04:28:29.809450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:29.797202 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="60aae71a-d64a-497b-b48f-f4767178434a" containerName="payload-pre-processing" Oct 30 04:28:29.809450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:29.801002 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" Oct 30 04:28:29.824448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:29.822285 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c"] Oct 30 04:28:29.844555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:29.834580 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ca7a81-dockercfg-h7mrf\"" Oct 30 04:28:29.844555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:29.837562 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ca7a81\"" Oct 30 04:28:29.917694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:29.917662 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vw2z\" (UniqueName: \"kubernetes.io/projected/60b4e089-0655-4e15-a46e-a50de07bb1ab-kube-api-access-5vw2z\") pod \"payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c\" (UID: \"60b4e089-0655-4e15-a46e-a50de07bb1ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" Oct 30 04:28:29.917814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:29.917781 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b4e089-0655-4e15-a46e-a50de07bb1ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c\" (UID: \"60b4e089-0655-4e15-a46e-a50de07bb1ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" Oct 30 04:28:30.018865 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.018838 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vw2z\" (UniqueName: \"kubernetes.io/projected/60b4e089-0655-4e15-a46e-a50de07bb1ab-kube-api-access-5vw2z\") pod \"payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c\" (UID: \"60b4e089-0655-4e15-a46e-a50de07bb1ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" Oct 30 04:28:30.019005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.018930 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b4e089-0655-4e15-a46e-a50de07bb1ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c\" (UID: \"60b4e089-0655-4e15-a46e-a50de07bb1ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" Oct 30 04:28:30.019636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.019613 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b4e089-0655-4e15-a46e-a50de07bb1ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c\" (UID: \"60b4e089-0655-4e15-a46e-a50de07bb1ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" Oct 30 04:28:30.042836 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.042790 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vw2z\" (UniqueName: \"kubernetes.io/projected/60b4e089-0655-4e15-a46e-a50de07bb1ab-kube-api-access-5vw2z\") pod \"payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c\" (UID: \"60b4e089-0655-4e15-a46e-a50de07bb1ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" Oct 30 04:28:30.140978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.140923 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" Oct 30 04:28:30.431185 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.431152 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:30.442991 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.442959 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c"] Oct 30 04:28:30.443356 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:28:30.443330 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b4e089_0655_4e15_a46e_a50de07bb1ab.slice/crio-a405871d00eca4e00d4bb2a51d61587c31caea677951747e8ae9c5b56b95f0c2 WatchSource:0}: Error finding container a405871d00eca4e00d4bb2a51d61587c31caea677951747e8ae9c5b56b95f0c2: Status 404 returned error can't find the container with id a405871d00eca4e00d4bb2a51d61587c31caea677951747e8ae9c5b56b95f0c2 Oct 30 04:28:30.899140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.899060 4793 generic.go:356] "Generic (PLEG): container finished" podID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerID="dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f" exitCode=0 Oct 30 04:28:30.899304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.899163 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" event={"ID":"7fc4700a-fc65-4c54-ab00-66df2dd352c5","Type":"ContainerDied","Data":"dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f"} Oct 30 04:28:30.899527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.899507 4793 scope.go:122] "RemoveContainer" containerID="dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f" Oct 30 04:28:30.901722 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.901691 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:28:30.901908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.901776 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:28:30.901908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.901826 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:28:30.902295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.902275 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" event={"ID":"60b4e089-0655-4e15-a46e-a50de07bb1ab","Type":"ContainerStarted","Data":"0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb"} Oct 30 04:28:30.902489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.902320 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" event={"ID":"60b4e089-0655-4e15-a46e-a50de07bb1ab","Type":"ContainerStarted","Data":"a405871d00eca4e00d4bb2a51d61587c31caea677951747e8ae9c5b56b95f0c2"} Oct 30 04:28:30.902489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.902463 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" Oct 30 04:28:30.977441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:30.977347 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:31.009964 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:31.009906 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" podStartSLOduration=2.009884938 podStartE2EDuration="2.009884938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:31.004042716 +0000 UTC m=+1834.289917494" watchObservedRunningTime="2026-10-30 04:28:31.009884938 +0000 UTC m=+1834.295759714" Oct 30 04:28:31.197968 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:31.197926 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:31.921332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:31.921296 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" event={"ID":"7fc4700a-fc65-4c54-ab00-66df2dd352c5","Type":"ContainerStarted","Data":"f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563"} Oct 30 04:28:32.136746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:32.136701 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:28:32.137088 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:32.136836 4793 patch_prober.go:29] interesting pod/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 04:28:32.137088 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:32.136887 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 04:28:32.275440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:32.269056 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:33.137523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:33.137486 4793 patch_prober.go:29] interesting pod/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 04:28:33.138055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:33.137544 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 04:28:33.891018 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:33.890954 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:34.137687 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:34.137653 4793 patch_prober.go:29] interesting pod/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 04:28:34.138086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:34.137703 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 04:28:35.030617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:35.030586 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p"] Oct 30 04:28:35.042884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:35.042858 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8094b640-xmf6p"] Oct 30 04:28:35.137568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:35.137529 4793 patch_prober.go:29] interesting pod/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 04:28:35.137733 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:35.137592 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 04:28:36.137937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:36.137893 4793 patch_prober.go:29] interesting pod/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 04:28:36.142451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:36.140805 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 04:28:36.176483 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:36.176448 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:36.847507 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:36.847473 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3149dd11-8250-4113-8498-88d836d6a5e1" path="/var/lib/kubelet/pods/3149dd11-8250-4113-8498-88d836d6a5e1/volumes" Oct 30 04:28:37.137001 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:37.136904 4793 patch_prober.go:29] interesting pod/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 04:28:37.137153 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:37.136992 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 04:28:37.425505 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:37.424690 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:37.656533 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:37.656492 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:38.142641 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:38.142615 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:28:38.142925 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:38.142902 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:28:38.143746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:38.143729 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:28:38.486544 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:38.486508 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:40.127050 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:40.126942 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:41.219764 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:28:41.219729 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:28:41.896459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:41.896397 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:41.924128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:41.924103 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" Oct 30 04:28:43.812957 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.812924 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c"] Oct 30 04:28:43.818920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.818900 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:43.841846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.833398 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/234dc359-6934-4120-b625-f6249ea314b6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:43.841846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.833627 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-metrics-certs\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:43.841846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.833703 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:43.841846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.833750 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h28mj\" (UniqueName: \"kubernetes.io/projected/234dc359-6934-4120-b625-f6249ea314b6-kube-api-access-h28mj\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:43.845450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.845408 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c"] Oct 30 04:28:43.851790 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.851771 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f7a36d-serving-cert\"" Oct 30 04:28:43.934360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.934336 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/234dc359-6934-4120-b625-f6249ea314b6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:43.934499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.934475 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-metrics-certs\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:43.934574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.934551 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:43.934634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.934602 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h28mj\" (UniqueName: \"kubernetes.io/projected/234dc359-6934-4120-b625-f6249ea314b6-kube-api-access-h28mj\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:43.934725 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:28:43.934702 4793 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-serving-cert: secret "maas-api-e2e-ipp-a-f7a36d-serving-cert" not found Oct 30 04:28:43.934810 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:28:43.934795 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-maas-api-tls podName:234dc359-6934-4120-b625-f6249ea314b6 nodeName:}" failed. No retries permitted until 2026-10-30 04:28:44.434771595 +0000 UTC m=+1847.720646369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-maas-api-tls") pod "maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" (UID: "234dc359-6934-4120-b625-f6249ea314b6") : secret "maas-api-e2e-ipp-a-f7a36d-serving-cert" not found Oct 30 04:28:43.935148 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.935121 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/234dc359-6934-4120-b625-f6249ea314b6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:43.937468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:43.937410 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-metrics-certs\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:44.023065 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:44.023038 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h28mj\" (UniqueName: \"kubernetes.io/projected/234dc359-6934-4120-b625-f6249ea314b6-kube-api-access-h28mj\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:44.070270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:44.070177 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:44.295774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:44.295738 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:44.439805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:44.439776 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:44.442297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:44.442265 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:44.736529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:44.736457 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:45.138754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:45.138714 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c"] Oct 30 04:28:45.833467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:45.833363 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:46.042324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:46.042287 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" event={"ID":"234dc359-6934-4120-b625-f6249ea314b6","Type":"ContainerStarted","Data":"2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab"} Oct 30 04:28:46.042548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:46.042346 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" event={"ID":"234dc359-6934-4120-b625-f6249ea314b6","Type":"ContainerStarted","Data":"7078375a9d94729865bb1f5f7880ca02a1f1911f741fa4292c26821e3452204b"} Oct 30 04:28:46.042548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:46.042383 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:46.124758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:46.124661 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" podStartSLOduration=2.76019222 podStartE2EDuration="3.124647019s" totalImagesPullingTime="364.454799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:46.119086357 +0000 UTC m=+1849.404961135" watchObservedRunningTime="2026-10-30 04:28:46.124647019 +0000 UTC m=+1849.410521795" Oct 30 04:28:47.636609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:47.636554 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:49.193905 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:49.193869 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:49.410117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:49.410074 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:49.871565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:49.871513 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:51.054392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:51.054213 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:51.273371 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:51.272378 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:52.059607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:52.059580 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:28:52.112828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:52.112794 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:53.607021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:53.606987 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:53.941188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:53.941158 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:54.483095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:54.483063 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:55.405591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:55.405556 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:56.119941 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.119901 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:56.337523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.337485 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:56.427407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.427380 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf"] Oct 30 04:28:56.431858 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.431833 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.449014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.448989 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3f5508-openshift-default-dockercfg-qm4jg\"" Oct 30 04:28:56.497549 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.497497 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf"] Oct 30 04:28:56.513458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.513399 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-envoy\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.513624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.513473 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-credential-socket\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.513624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.513499 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-certs\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.513624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.513542 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istiod-ca-cert\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.513624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.513571 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-data\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.513877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.513642 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-token\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.513877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.513666 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-podinfo\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.513877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.513711 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-socket\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.513877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.513733 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjkd5\" (UniqueName: \"kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-kube-api-access-zjkd5\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.590947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.590908 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:56.615075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615044 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-token\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.615244 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615090 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-podinfo\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.615244 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615153 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-socket\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.615244 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615178 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjkd5\" (UniqueName: \"kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-kube-api-access-zjkd5\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.615448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615303 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-envoy\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.615448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615354 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-credential-socket\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.615448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615385 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-certs\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.615613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615467 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istiod-ca-cert\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.615613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615516 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-data\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.615716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615640 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-socket\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.615716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615693 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-credential-socket\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.615820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615774 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-certs\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.615895 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.615874 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-data\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.616254 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.616230 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istiod-ca-cert\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.617968 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.617947 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-envoy\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.618286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.618266 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-podinfo\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.683496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.672903 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjkd5\" (UniqueName: \"kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-kube-api-access-zjkd5\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.686329 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.686297 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-token\") pod \"e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.754939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.754662 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:56.896489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:56.879681 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:57.220690 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:28:57.220659 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4b2377_59fe_4459_ae4f_7cbbe753a8cf.slice/crio-f0a314bc474e0da8f739265337fcdcb0a5ce2164bd3e951045f4aff3cc61ef6e WatchSource:0}: Error finding container f0a314bc474e0da8f739265337fcdcb0a5ce2164bd3e951045f4aff3cc61ef6e: Status 404 returned error can't find the container with id f0a314bc474e0da8f739265337fcdcb0a5ce2164bd3e951045f4aff3cc61ef6e Oct 30 04:28:57.221568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:57.221541 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf"] Oct 30 04:28:57.223617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:57.223586 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:28:57.223681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:57.223646 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:28:57.223681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:57.223676 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:28:58.110084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:58.110053 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:28:58.153149 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:58.153116 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" event={"ID":"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf","Type":"ContainerStarted","Data":"42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04"} Oct 30 04:28:58.153315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:58.153170 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" event={"ID":"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf","Type":"ContainerStarted","Data":"f0a314bc474e0da8f739265337fcdcb0a5ce2164bd3e951045f4aff3cc61ef6e"} Oct 30 04:28:58.230750 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:58.230692 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" podStartSLOduration=2.230677688 podStartE2EDuration="2.230677688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:28:58.216562535 +0000 UTC m=+1861.502437310" watchObservedRunningTime="2026-10-30 04:28:58.230677688 +0000 UTC m=+1861.516552463" Oct 30 04:28:58.784480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:58.777833 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:58.792704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:58.792675 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:59.188824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:59.186875 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:59.188824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:59.188498 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:28:59.937190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:28:59.937154 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:00.739937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:00.739906 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:00.954059 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:00.954016 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:01.314751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:01.314715 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:02.419210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:02.419176 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:02.662004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:02.661965 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:04.225760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:04.225719 4793 generic.go:356] "Generic (PLEG): container finished" podID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerID="e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1" exitCode=0 Oct 30 04:29:04.226328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:04.225810 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" event={"ID":"40b57c75-ef43-4572-9e9c-2c56102a6d5b","Type":"ContainerDied","Data":"e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1"} Oct 30 04:29:04.226328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:04.226261 4793 scope.go:122] "RemoveContainer" containerID="e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1" Oct 30 04:29:04.227897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:04.227865 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:29:04.228004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:04.227936 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:29:04.228004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:04.227990 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:29:04.929631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:04.929594 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:05.237372 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:05.237277 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" event={"ID":"40b57c75-ef43-4572-9e9c-2c56102a6d5b","Type":"ContainerStarted","Data":"068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2"} Oct 30 04:29:05.720645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:05.720618 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:29:05.926147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:05.926109 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:29:05.927531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:05.927500 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 04:29:05.927665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:05.927569 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 04:29:06.805843 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:06.805799 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:06.926010 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:06.925979 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 04:29:06.926190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:06.926028 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 04:29:07.926256 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:07.926224 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 04:29:07.926692 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:07.926281 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 04:29:08.550875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:08.550836 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:08.926954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:08.926919 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 04:29:08.927343 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:08.926991 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 04:29:09.186352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:09.186255 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:09.926652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:09.926613 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Oct 30 04:29:09.926834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:09.926683 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Oct 30 04:29:10.732433 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:10.732376 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:10.931878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:10.931850 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:29:10.932868 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:10.932843 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:29:11.219213 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:29:11.219179 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:29:12.558057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:12.558027 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:14.132571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.132514 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:14.133245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.133221 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq"] Oct 30 04:29:14.133567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.133516 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" podUID="8a2c9185-072d-4655-806d-7434d227252c" containerName="payload-processing" containerID="cri-o://448e2b84969e8ecdabd264162eda26314346ca1b3f17cda262153900aa0c9de1" gracePeriod=30 Oct 30 04:29:14.336345 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.336309 4793 generic.go:356] "Generic (PLEG): container finished" podID="8a2c9185-072d-4655-806d-7434d227252c" containerID="448e2b84969e8ecdabd264162eda26314346ca1b3f17cda262153900aa0c9de1" exitCode=0 Oct 30 04:29:14.336621 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.336601 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" event={"ID":"8a2c9185-072d-4655-806d-7434d227252c","Type":"ContainerDied","Data":"448e2b84969e8ecdabd264162eda26314346ca1b3f17cda262153900aa0c9de1"} Oct 30 04:29:14.346352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.345818 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:14.528440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.527993 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" Oct 30 04:29:14.570154 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.570117 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:14.625775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.625745 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a2c9185-072d-4655-806d-7434d227252c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2c9185-072d-4655-806d-7434d227252c-plugins-config-volume\") pod \"8a2c9185-072d-4655-806d-7434d227252c\" (UID: \"8a2c9185-072d-4655-806d-7434d227252c\") " Oct 30 04:29:14.625945 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.625838 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a2c9185-072d-4655-806d-7434d227252c-kube-api-access-tn4rp\" (UniqueName: \"kubernetes.io/projected/8a2c9185-072d-4655-806d-7434d227252c-kube-api-access-tn4rp\") pod \"8a2c9185-072d-4655-806d-7434d227252c\" (UID: \"8a2c9185-072d-4655-806d-7434d227252c\") " Oct 30 04:29:14.626123 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.626102 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a2c9185-072d-4655-806d-7434d227252c-plugins-config-volume" pod "8a2c9185-072d-4655-806d-7434d227252c" (UID: "8a2c9185-072d-4655-806d-7434d227252c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:29:14.626321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.626306 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a2c9185-072d-4655-806d-7434d227252c-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:14.627854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.627833 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2c9185-072d-4655-806d-7434d227252c-kube-api-access-tn4rp" pod "8a2c9185-072d-4655-806d-7434d227252c" (UID: "8a2c9185-072d-4655-806d-7434d227252c"). InnerVolumeSpecName "kube-api-access-tn4rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:14.728111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.728078 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4rp\" (UniqueName: \"kubernetes.io/projected/8a2c9185-072d-4655-806d-7434d227252c-kube-api-access-tn4rp\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:14.961522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:14.961489 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:15.344774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:15.344747 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" Oct 30 04:29:15.345245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:15.344791 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq" event={"ID":"8a2c9185-072d-4655-806d-7434d227252c","Type":"ContainerDied","Data":"517169f9f75739526331882bdcbf19739701746df9f3631ffa8fa3c66c4cdc1e"} Oct 30 04:29:15.345245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:15.344858 4793 scope.go:122] "RemoveContainer" containerID="448e2b84969e8ecdabd264162eda26314346ca1b3f17cda262153900aa0c9de1" Oct 30 04:29:15.399222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:15.399191 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq"] Oct 30 04:29:15.411387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:15.411364 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf4238-6bfd8864cb-9v8jq"] Oct 30 04:29:15.511614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:15.511582 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:16.848516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:16.848481 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a2c9185-072d-4655-806d-7434d227252c" path="/var/lib/kubelet/pods/8a2c9185-072d-4655-806d-7434d227252c/volumes" Oct 30 04:29:17.362578 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:17.362541 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:17.628286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:17.628116 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:17.857486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:17.857451 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:19.221475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:19.221441 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:20.610132 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:20.610095 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:20.936260 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:20.936222 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:21.526271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:21.526235 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:21.756000 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:21.755957 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:21.800496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:21.791738 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25"] Oct 30 04:29:21.800496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:21.792003 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" containerID="cri-o://1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707" gracePeriod=30 Oct 30 04:29:22.034267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:22.033496 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:22.783860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:22.783797 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:23.360824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:23.360778 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:24.502434 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:24.502383 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:24.803200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:24.803082 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:25.200183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:25.200133 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:25.424243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:25.424198 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:26.869792 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:26.869744 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:26.905155 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:26.905122 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r"] Oct 30 04:29:26.905364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:26.905342 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" podUID="304cae42-e227-4708-a6c4-20c2b7cc9768" containerName="payload-processing" containerID="cri-o://33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b" gracePeriod=30 Oct 30 04:29:27.090821 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.090793 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:29:27.120214 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.120140 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:27.258448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.231253 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-podinfo\") pod \"f62df4ee-f0ca-4788-8723-1619ec96c657\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " Oct 30 04:29:27.258448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.231353 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-token\" (UniqueName: \"kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-token\") pod \"f62df4ee-f0ca-4788-8723-1619ec96c657\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " Oct 30 04:29:27.258448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.231442 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f62df4ee-f0ca-4788-8723-1619ec96c657-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62df4ee-f0ca-4788-8723-1619ec96c657-istiod-ca-cert\") pod \"f62df4ee-f0ca-4788-8723-1619ec96c657\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " Oct 30 04:29:27.258448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.231496 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-envoy\") pod \"f62df4ee-f0ca-4788-8723-1619ec96c657\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " Oct 30 04:29:27.258448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.231537 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-data\") pod \"f62df4ee-f0ca-4788-8723-1619ec96c657\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " Oct 30 04:29:27.258448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.231583 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-credential-socket\") pod \"f62df4ee-f0ca-4788-8723-1619ec96c657\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " Oct 30 04:29:27.258448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.231619 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-kube-api-access-79cq7\" (UniqueName: \"kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-kube-api-access-79cq7\") pod \"f62df4ee-f0ca-4788-8723-1619ec96c657\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " Oct 30 04:29:27.258448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.231676 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-certs\") pod \"f62df4ee-f0ca-4788-8723-1619ec96c657\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " Oct 30 04:29:27.258448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.231720 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-socket\") pod \"f62df4ee-f0ca-4788-8723-1619ec96c657\" (UID: \"f62df4ee-f0ca-4788-8723-1619ec96c657\") " Oct 30 04:29:27.282516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.278579 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-podinfo" pod "f62df4ee-f0ca-4788-8723-1619ec96c657" (UID: "f62df4ee-f0ca-4788-8723-1619ec96c657"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:29:27.282516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.279898 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-certs" pod "f62df4ee-f0ca-4788-8723-1619ec96c657" (UID: "f62df4ee-f0ca-4788-8723-1619ec96c657"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:27.282516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.280084 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-credential-socket" pod "f62df4ee-f0ca-4788-8723-1619ec96c657" (UID: "f62df4ee-f0ca-4788-8723-1619ec96c657"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:27.282516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.280449 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f62df4ee-f0ca-4788-8723-1619ec96c657-istiod-ca-cert" pod "f62df4ee-f0ca-4788-8723-1619ec96c657" (UID: "f62df4ee-f0ca-4788-8723-1619ec96c657"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:29:27.283316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.283277 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-socket" pod "f62df4ee-f0ca-4788-8723-1619ec96c657" (UID: "f62df4ee-f0ca-4788-8723-1619ec96c657"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:27.284816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.284777 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-data" pod "f62df4ee-f0ca-4788-8723-1619ec96c657" (UID: "f62df4ee-f0ca-4788-8723-1619ec96c657"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:27.303240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.302928 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" Oct 30 04:29:27.309715 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.309662 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-envoy" pod "f62df4ee-f0ca-4788-8723-1619ec96c657" (UID: "f62df4ee-f0ca-4788-8723-1619ec96c657"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:27.311933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.311901 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-kube-api-access-79cq7" pod "f62df4ee-f0ca-4788-8723-1619ec96c657" (UID: "f62df4ee-f0ca-4788-8723-1619ec96c657"). InnerVolumeSpecName "kube-api-access-79cq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:27.333839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.333669 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f62df4ee-f0ca-4788-8723-1619ec96c657-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:27.333839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.333697 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:27.333839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.333711 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:27.333839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.333725 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:27.333839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.333739 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79cq7\" (UniqueName: \"kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-kube-api-access-79cq7\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:27.333839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.333753 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:27.333839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.333768 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f62df4ee-f0ca-4788-8723-1619ec96c657-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:27.333839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.333780 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:27.337644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.337612 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-token" pod "f62df4ee-f0ca-4788-8723-1619ec96c657" (UID: "f62df4ee-f0ca-4788-8723-1619ec96c657"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:27.351732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.351502 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:27.435257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.435223 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304cae42-e227-4708-a6c4-20c2b7cc9768-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/304cae42-e227-4708-a6c4-20c2b7cc9768-plugins-config-volume\") pod \"304cae42-e227-4708-a6c4-20c2b7cc9768\" (UID: \"304cae42-e227-4708-a6c4-20c2b7cc9768\") " Oct 30 04:29:27.435457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.435295 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304cae42-e227-4708-a6c4-20c2b7cc9768-kube-api-access-wjm87\" (UniqueName: \"kubernetes.io/projected/304cae42-e227-4708-a6c4-20c2b7cc9768-kube-api-access-wjm87\") pod \"304cae42-e227-4708-a6c4-20c2b7cc9768\" (UID: \"304cae42-e227-4708-a6c4-20c2b7cc9768\") " Oct 30 04:29:27.435685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.435656 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304cae42-e227-4708-a6c4-20c2b7cc9768-plugins-config-volume" pod "304cae42-e227-4708-a6c4-20c2b7cc9768" (UID: "304cae42-e227-4708-a6c4-20c2b7cc9768"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:29:27.435879 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.435861 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f62df4ee-f0ca-4788-8723-1619ec96c657-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:27.435961 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.435883 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/304cae42-e227-4708-a6c4-20c2b7cc9768-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:27.437568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.437545 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304cae42-e227-4708-a6c4-20c2b7cc9768-kube-api-access-wjm87" pod "304cae42-e227-4708-a6c4-20c2b7cc9768" (UID: "304cae42-e227-4708-a6c4-20c2b7cc9768"). InnerVolumeSpecName "kube-api-access-wjm87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:27.438582 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.438560 4793 generic.go:356] "Generic (PLEG): container finished" podID="304cae42-e227-4708-a6c4-20c2b7cc9768" containerID="33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b" exitCode=0 Oct 30 04:29:27.438678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.438626 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" Oct 30 04:29:27.438678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.438644 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" event={"ID":"304cae42-e227-4708-a6c4-20c2b7cc9768","Type":"ContainerDied","Data":"33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b"} Oct 30 04:29:27.438794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.438702 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r" event={"ID":"304cae42-e227-4708-a6c4-20c2b7cc9768","Type":"ContainerDied","Data":"60f2887d6e98cd7319381298298ca489d0dec0ae699aea202375a26c670dba65"} Oct 30 04:29:27.438794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.438732 4793 scope.go:122] "RemoveContainer" containerID="33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b" Oct 30 04:29:27.440490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.440467 4793 generic.go:356] "Generic (PLEG): container finished" podID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerID="1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707" exitCode=0 Oct 30 04:29:27.440619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.440605 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" event={"ID":"f62df4ee-f0ca-4788-8723-1619ec96c657","Type":"ContainerDied","Data":"1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707"} Oct 30 04:29:27.440663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.440641 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" event={"ID":"f62df4ee-f0ca-4788-8723-1619ec96c657","Type":"ContainerDied","Data":"65369617012c0ca8aaa75a1ea4c4c0034441582baa4ba96652792f25172325b6"} Oct 30 04:29:27.440713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.440702 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25" Oct 30 04:29:27.449160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.449137 4793 scope.go:122] "RemoveContainer" containerID="33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b" Oct 30 04:29:27.449505 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:29:27.449438 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b\": container with ID starting with 33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b not found: ID does not exist" containerID="33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b" Oct 30 04:29:27.449505 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.449476 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b"} err="failed to get container status \"33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b\": rpc error: code = NotFound desc = could not find container \"33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b\": container with ID starting with 33431499d1040f2c34a3888bb7f32917487b2a34e82fd98c55a02f15bdb2388b not found: ID does not exist" Oct 30 04:29:27.449505 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.449499 4793 scope.go:122] "RemoveContainer" containerID="1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707" Oct 30 04:29:27.460102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.460076 4793 scope.go:122] "RemoveContainer" containerID="bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98" Oct 30 04:29:27.469995 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.469979 4793 scope.go:122] "RemoveContainer" containerID="1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707" Oct 30 04:29:27.470301 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:29:27.470275 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707\": container with ID starting with 1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707 not found: ID does not exist" containerID="1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707" Oct 30 04:29:27.470395 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.470311 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707"} err="failed to get container status \"1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707\": rpc error: code = NotFound desc = could not find container \"1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707\": container with ID starting with 1197e409bb4eb203c5d174974565952cba3a0e74423933621fceadf15dd09707 not found: ID does not exist" Oct 30 04:29:27.470395 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.470334 4793 scope.go:122] "RemoveContainer" containerID="bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98" Oct 30 04:29:27.470763 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:29:27.470738 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98\": container with ID starting with bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98 not found: ID does not exist" containerID="bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98" Oct 30 04:29:27.470830 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.470763 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98"} err="failed to get container status \"bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98\": rpc error: code = NotFound desc = could not find container \"bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98\": container with ID starting with bc3cf9fb7748a12d1c4eaf720c927d2698fa514c6e9d7960bfccd3dccb61ec98 not found: ID does not exist" Oct 30 04:29:27.495858 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.495835 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r"] Oct 30 04:29:27.501286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.501259 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6dcea5-fd46bc8f8-cd55r"] Oct 30 04:29:27.531836 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.531814 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25"] Oct 30 04:29:27.537124 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.537098 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjm87\" (UniqueName: \"kubernetes.io/projected/304cae42-e227-4708-a6c4-20c2b7cc9768-kube-api-access-wjm87\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:27.538099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.538081 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf4238-openshift-default-df464f8f-5tr25"] Oct 30 04:29:27.810383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:27.810300 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:28.847935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:28.847900 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304cae42-e227-4708-a6c4-20c2b7cc9768" path="/var/lib/kubelet/pods/304cae42-e227-4708-a6c4-20c2b7cc9768/volumes" Oct 30 04:29:28.848440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:28.848408 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" path="/var/lib/kubelet/pods/f62df4ee-f0ca-4788-8723-1619ec96c657/volumes" Oct 30 04:29:29.708942 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:29.708900 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:30.025638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:30.025551 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:31.772691 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:31.772651 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:33.081294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.081263 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b"] Oct 30 04:29:33.081782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.081501 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" podUID="4ea80f40-4200-4e8c-9cfe-8e030f7a39f3" containerName="maas-api" containerID="cri-o://0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a" gracePeriod=30 Oct 30 04:29:33.267268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.267097 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:33.417438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.417394 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:29:33.489357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.489327 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:33.496284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.496253 4793 generic.go:356] "Generic (PLEG): container finished" podID="4ea80f40-4200-4e8c-9cfe-8e030f7a39f3" containerID="0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a" exitCode=0 Oct 30 04:29:33.496462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.496300 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" event={"ID":"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3","Type":"ContainerDied","Data":"0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a"} Oct 30 04:29:33.496462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.496314 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" Oct 30 04:29:33.496462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.496351 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b" event={"ID":"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3","Type":"ContainerDied","Data":"d58aa7b10090bebf2a9aa7f25cdf6b6554f9f96d1029b0af71542154483eaf1f"} Oct 30 04:29:33.496462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.496384 4793 scope.go:122] "RemoveContainer" containerID="0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a" Oct 30 04:29:33.515263 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.515243 4793 scope.go:122] "RemoveContainer" containerID="0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a" Oct 30 04:29:33.515577 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:29:33.515555 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a\": container with ID starting with 0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a not found: ID does not exist" containerID="0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a" Oct 30 04:29:33.515680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.515584 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a"} err="failed to get container status \"0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a\": rpc error: code = NotFound desc = could not find container \"0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a\": container with ID starting with 0c2960a95e9f0328b1006d642fba314fc5b10d885db88baba2c28da94ee7d74a not found: ID does not exist" Oct 30 04:29:33.515680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.515633 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-metrics-certs\") pod \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " Oct 30 04:29:33.515816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.515802 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-openshift-service-ca\") pod \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " Oct 30 04:29:33.515871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.515842 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-maas-api-tls\") pod \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " Oct 30 04:29:33.515921 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.515875 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-kube-api-access-jb5sw\" (UniqueName: \"kubernetes.io/projected/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-kube-api-access-jb5sw\") pod \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\" (UID: \"4ea80f40-4200-4e8c-9cfe-8e030f7a39f3\") " Oct 30 04:29:33.516191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.516164 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-openshift-service-ca" pod "4ea80f40-4200-4e8c-9cfe-8e030f7a39f3" (UID: "4ea80f40-4200-4e8c-9cfe-8e030f7a39f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:29:33.516872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.516844 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:33.517669 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.517644 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-metrics-certs" pod "4ea80f40-4200-4e8c-9cfe-8e030f7a39f3" (UID: "4ea80f40-4200-4e8c-9cfe-8e030f7a39f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:29:33.517937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.517920 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-kube-api-access-jb5sw" pod "4ea80f40-4200-4e8c-9cfe-8e030f7a39f3" (UID: "4ea80f40-4200-4e8c-9cfe-8e030f7a39f3"). InnerVolumeSpecName "kube-api-access-jb5sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:33.518097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.518078 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-maas-api-tls" pod "4ea80f40-4200-4e8c-9cfe-8e030f7a39f3" (UID: "4ea80f40-4200-4e8c-9cfe-8e030f7a39f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:29:33.647498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.640471 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:33.647498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.640510 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:33.647498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.640525 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb5sw\" (UniqueName: \"kubernetes.io/projected/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3-kube-api-access-jb5sw\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:33.789521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.789479 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:33.849498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.849468 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b"] Oct 30 04:29:33.860696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:33.860668 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-baa474-5fdfdcf79c-nff7b"] Oct 30 04:29:34.135008 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:34.134961 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:34.846962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:34.846930 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea80f40-4200-4e8c-9cfe-8e030f7a39f3" path="/var/lib/kubelet/pods/4ea80f40-4200-4e8c-9cfe-8e030f7a39f3/volumes" Oct 30 04:29:35.469605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.469529 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:35.695441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.694899 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt"] Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.695792 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.695817 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.695839 4793 cpu_manager.go:414] "containerMap: removing container" podUID="304cae42-e227-4708-a6c4-20c2b7cc9768" containerName="payload-processing" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.695847 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304cae42-e227-4708-a6c4-20c2b7cc9768" containerName="payload-processing" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.695873 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.695880 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.695917 4793 cpu_manager.go:414] "containerMap: removing container" podUID="8a2c9185-072d-4655-806d-7434d227252c" containerName="payload-processing" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.695925 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a2c9185-072d-4655-806d-7434d227252c" containerName="payload-processing" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.695935 4793 cpu_manager.go:414] "containerMap: removing container" podUID="4ea80f40-4200-4e8c-9cfe-8e030f7a39f3" containerName="maas-api" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.695943 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea80f40-4200-4e8c-9cfe-8e030f7a39f3" containerName="maas-api" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.696081 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a2c9185-072d-4655-806d-7434d227252c" containerName="payload-processing" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.696092 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea80f40-4200-4e8c-9cfe-8e030f7a39f3" containerName="maas-api" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.696106 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" Oct 30 04:29:35.699436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.696116 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="304cae42-e227-4708-a6c4-20c2b7cc9768" containerName="payload-processing" Oct 30 04:29:35.725492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.715397 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:35.748984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.748953 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt"] Oct 30 04:29:35.749304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.749281 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" Oct 30 04:29:35.770890 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.770861 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f7a36d-dockercfg-xsvbc\"" Oct 30 04:29:35.771013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.770947 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f7a36d\"" Oct 30 04:29:35.815402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.815371 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btr5v\" (UniqueName: \"kubernetes.io/projected/0f61929c-96a1-4073-8f8c-acb25ad6d040-kube-api-access-btr5v\") pod \"payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt\" (UID: \"0f61929c-96a1-4073-8f8c-acb25ad6d040\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" Oct 30 04:29:35.815558 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.815523 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f61929c-96a1-4073-8f8c-acb25ad6d040-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt\" (UID: \"0f61929c-96a1-4073-8f8c-acb25ad6d040\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" Oct 30 04:29:35.917103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.917073 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f61929c-96a1-4073-8f8c-acb25ad6d040-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt\" (UID: \"0f61929c-96a1-4073-8f8c-acb25ad6d040\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" Oct 30 04:29:35.917385 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.917229 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btr5v\" (UniqueName: \"kubernetes.io/projected/0f61929c-96a1-4073-8f8c-acb25ad6d040-kube-api-access-btr5v\") pod \"payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt\" (UID: \"0f61929c-96a1-4073-8f8c-acb25ad6d040\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" Oct 30 04:29:35.917772 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.917752 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f61929c-96a1-4073-8f8c-acb25ad6d040-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt\" (UID: \"0f61929c-96a1-4073-8f8c-acb25ad6d040\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" Oct 30 04:29:35.942509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.942478 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btr5v\" (UniqueName: \"kubernetes.io/projected/0f61929c-96a1-4073-8f8c-acb25ad6d040-kube-api-access-btr5v\") pod \"payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt\" (UID: \"0f61929c-96a1-4073-8f8c-acb25ad6d040\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" Oct 30 04:29:35.944313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.944289 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb"] Oct 30 04:29:35.944985 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.944968 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62df4ee-f0ca-4788-8723-1619ec96c657" containerName="istio-proxy" Oct 30 04:29:35.947383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.947366 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" Oct 30 04:29:35.968794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:35.968765 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb"] Oct 30 04:29:36.018130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.018051 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8a7afc-f51a-49ce-8837-0d5f39b03600-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb\" (UID: \"aa8a7afc-f51a-49ce-8837-0d5f39b03600\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" Oct 30 04:29:36.018269 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.018180 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-www45\" (UniqueName: \"kubernetes.io/projected/aa8a7afc-f51a-49ce-8837-0d5f39b03600-kube-api-access-www45\") pod \"payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb\" (UID: \"aa8a7afc-f51a-49ce-8837-0d5f39b03600\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" Oct 30 04:29:36.072840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.072810 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" Oct 30 04:29:36.119458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.119407 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8a7afc-f51a-49ce-8837-0d5f39b03600-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb\" (UID: \"aa8a7afc-f51a-49ce-8837-0d5f39b03600\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" Oct 30 04:29:36.119655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.119511 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-www45\" (UniqueName: \"kubernetes.io/projected/aa8a7afc-f51a-49ce-8837-0d5f39b03600-kube-api-access-www45\") pod \"payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb\" (UID: \"aa8a7afc-f51a-49ce-8837-0d5f39b03600\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" Oct 30 04:29:36.120250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.120218 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8a7afc-f51a-49ce-8837-0d5f39b03600-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb\" (UID: \"aa8a7afc-f51a-49ce-8837-0d5f39b03600\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" Oct 30 04:29:36.150783 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.150749 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-www45\" (UniqueName: \"kubernetes.io/projected/aa8a7afc-f51a-49ce-8837-0d5f39b03600-kube-api-access-www45\") pod \"payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb\" (UID: \"aa8a7afc-f51a-49ce-8837-0d5f39b03600\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" Oct 30 04:29:36.267437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.267390 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" Oct 30 04:29:36.362703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.362676 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:36.379460 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:29:36.379140 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f61929c_96a1_4073_8f8c_acb25ad6d040.slice/crio-a9b7bf90c244324807e6807a3f307eb73998bc9c31302e746753bf0e3cac2ea9 WatchSource:0}: Error finding container a9b7bf90c244324807e6807a3f307eb73998bc9c31302e746753bf0e3cac2ea9: Status 404 returned error can't find the container with id a9b7bf90c244324807e6807a3f307eb73998bc9c31302e746753bf0e3cac2ea9 Oct 30 04:29:36.380719 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.380697 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt"] Oct 30 04:29:36.524979 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.524852 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" event={"ID":"0f61929c-96a1-4073-8f8c-acb25ad6d040","Type":"ContainerStarted","Data":"ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51"} Oct 30 04:29:36.524979 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.524923 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" event={"ID":"0f61929c-96a1-4073-8f8c-acb25ad6d040","Type":"ContainerStarted","Data":"a9b7bf90c244324807e6807a3f307eb73998bc9c31302e746753bf0e3cac2ea9"} Oct 30 04:29:36.524979 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.524954 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" Oct 30 04:29:36.573024 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.572968 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" podStartSLOduration=1.572953086 podStartE2EDuration="1.572953086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:36.569994248 +0000 UTC m=+1899.855869023" watchObservedRunningTime="2026-10-30 04:29:36.572953086 +0000 UTC m=+1899.858827860" Oct 30 04:29:36.593678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.593382 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb"] Oct 30 04:29:36.593918 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:29:36.593885 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8a7afc_f51a_49ce_8837_0d5f39b03600.slice/crio-491c29fb53a98d884fa8b710266cf7669f01c0e42bdf01fd98dffeee2b43a5e5 WatchSource:0}: Error finding container 491c29fb53a98d884fa8b710266cf7669f01c0e42bdf01fd98dffeee2b43a5e5: Status 404 returned error can't find the container with id 491c29fb53a98d884fa8b710266cf7669f01c0e42bdf01fd98dffeee2b43a5e5 Oct 30 04:29:36.594937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:36.594858 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:37.195080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:37.195042 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:37.534623 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:37.534532 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" event={"ID":"aa8a7afc-f51a-49ce-8837-0d5f39b03600","Type":"ContainerStarted","Data":"8605fe7a53f0327cd743014d10790c3cbd331b4b5eb583784f814f71725a0b6f"} Oct 30 04:29:37.534623 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:37.534588 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" event={"ID":"aa8a7afc-f51a-49ce-8837-0d5f39b03600","Type":"ContainerStarted","Data":"491c29fb53a98d884fa8b710266cf7669f01c0e42bdf01fd98dffeee2b43a5e5"} Oct 30 04:29:37.534623 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:37.534613 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" Oct 30 04:29:37.638013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:37.637964 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:38.509292 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:38.509262 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:38.733814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:38.733780 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:40.329355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:40.329315 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:40.547604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:40.547570 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:40.877772 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:40.877728 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:41.109325 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:41.109288 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:41.219267 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:29:41.219236 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:29:41.470861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:41.470734 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" podStartSLOduration=6.4707154 podStartE2EDuration="6.4707154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:37.578574997 +0000 UTC m=+1900.864449771" watchObservedRunningTime="2026-10-30 04:29:41.4707154 +0000 UTC m=+1904.756590177" Oct 30 04:29:41.473883 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:41.473859 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc"] Oct 30 04:29:41.474159 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:41.474127 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerName="istio-proxy" containerID="cri-o://ebc1654ad9e3bf3fd29e6cc29229738b62a125f1e04d021227259041aa16fc84" gracePeriod=30 Oct 30 04:29:41.606137 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:41.606084 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:42.462465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.459582 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:42.526299 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.526253 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm"] Oct 30 04:29:42.527097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.526893 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" podUID="df5168c8-44f4-4ef6-84de-1a9437aca7a2" containerName="maas-api" containerID="cri-o://0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1" gracePeriod=30 Oct 30 04:29:42.695030 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.694964 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:42.719731 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.717389 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c"] Oct 30 04:29:42.719731 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.717666 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" podUID="60b4e089-0655-4e15-a46e-a50de07bb1ab" containerName="payload-processing" containerID="cri-o://0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb" gracePeriod=30 Oct 30 04:29:42.870795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.870494 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:29:42.948977 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.948936 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-maas-api-tls\") pod \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " Oct 30 04:29:42.963472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.950848 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5168c8-44f4-4ef6-84de-1a9437aca7a2-kube-api-access-dw8hz\" (UniqueName: \"kubernetes.io/projected/df5168c8-44f4-4ef6-84de-1a9437aca7a2-kube-api-access-dw8hz\") pod \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " Oct 30 04:29:42.963472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.950956 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df5168c8-44f4-4ef6-84de-1a9437aca7a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5168c8-44f4-4ef6-84de-1a9437aca7a2-openshift-service-ca\") pod \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " Oct 30 04:29:42.963472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.951057 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-metrics-certs\") pod \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\" (UID: \"df5168c8-44f4-4ef6-84de-1a9437aca7a2\") " Oct 30 04:29:42.973160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.963783 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5168c8-44f4-4ef6-84de-1a9437aca7a2-openshift-service-ca" pod "df5168c8-44f4-4ef6-84de-1a9437aca7a2" (UID: "df5168c8-44f4-4ef6-84de-1a9437aca7a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:29:42.973160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.950591 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:42.973605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.973522 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-metrics-certs" pod "df5168c8-44f4-4ef6-84de-1a9437aca7a2" (UID: "df5168c8-44f4-4ef6-84de-1a9437aca7a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:29:42.987474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.976777 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5168c8-44f4-4ef6-84de-1a9437aca7a2-kube-api-access-dw8hz" pod "df5168c8-44f4-4ef6-84de-1a9437aca7a2" (UID: "df5168c8-44f4-4ef6-84de-1a9437aca7a2"). InnerVolumeSpecName "kube-api-access-dw8hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:42.987474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:42.976858 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-maas-api-tls" pod "df5168c8-44f4-4ef6-84de-1a9437aca7a2" (UID: "df5168c8-44f4-4ef6-84de-1a9437aca7a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:29:43.053042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.053002 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:43.053042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.053043 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw8hz\" (UniqueName: \"kubernetes.io/projected/df5168c8-44f4-4ef6-84de-1a9437aca7a2-kube-api-access-dw8hz\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:43.053228 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.053059 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df5168c8-44f4-4ef6-84de-1a9437aca7a2-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:43.053228 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.053074 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df5168c8-44f4-4ef6-84de-1a9437aca7a2-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:43.228310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.228216 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:43.239471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.235367 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" Oct 30 04:29:43.385867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.385828 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60b4e089-0655-4e15-a46e-a50de07bb1ab-kube-api-access-5vw2z\" (UniqueName: \"kubernetes.io/projected/60b4e089-0655-4e15-a46e-a50de07bb1ab-kube-api-access-5vw2z\") pod \"60b4e089-0655-4e15-a46e-a50de07bb1ab\" (UID: \"60b4e089-0655-4e15-a46e-a50de07bb1ab\") " Oct 30 04:29:43.386698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.386650 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60b4e089-0655-4e15-a46e-a50de07bb1ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b4e089-0655-4e15-a46e-a50de07bb1ab-plugins-config-volume\") pod \"60b4e089-0655-4e15-a46e-a50de07bb1ab\" (UID: \"60b4e089-0655-4e15-a46e-a50de07bb1ab\") " Oct 30 04:29:43.387853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.387810 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b4e089-0655-4e15-a46e-a50de07bb1ab-plugins-config-volume" pod "60b4e089-0655-4e15-a46e-a50de07bb1ab" (UID: "60b4e089-0655-4e15-a46e-a50de07bb1ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:29:43.390614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.390588 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b4e089-0655-4e15-a46e-a50de07bb1ab-kube-api-access-5vw2z" pod "60b4e089-0655-4e15-a46e-a50de07bb1ab" (UID: "60b4e089-0655-4e15-a46e-a50de07bb1ab"). InnerVolumeSpecName "kube-api-access-5vw2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:43.488461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.488354 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b4e089-0655-4e15-a46e-a50de07bb1ab-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:43.488461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.488387 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vw2z\" (UniqueName: \"kubernetes.io/projected/60b4e089-0655-4e15-a46e-a50de07bb1ab-kube-api-access-5vw2z\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:43.609534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.609388 4793 generic.go:356] "Generic (PLEG): container finished" podID="df5168c8-44f4-4ef6-84de-1a9437aca7a2" containerID="0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1" exitCode=0 Oct 30 04:29:43.609534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.609477 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" Oct 30 04:29:43.609915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.609715 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" event={"ID":"df5168c8-44f4-4ef6-84de-1a9437aca7a2","Type":"ContainerDied","Data":"0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1"} Oct 30 04:29:43.609915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.609768 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm" event={"ID":"df5168c8-44f4-4ef6-84de-1a9437aca7a2","Type":"ContainerDied","Data":"4e0012725177114b703cae9cdb80585e725548bac2d19379c7271cfd773a2713"} Oct 30 04:29:43.609915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.609798 4793 scope.go:122] "RemoveContainer" containerID="0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1" Oct 30 04:29:43.612017 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.611994 4793 generic.go:356] "Generic (PLEG): container finished" podID="60b4e089-0655-4e15-a46e-a50de07bb1ab" containerID="0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb" exitCode=0 Oct 30 04:29:43.612266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.612044 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" Oct 30 04:29:43.612388 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.612063 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" event={"ID":"60b4e089-0655-4e15-a46e-a50de07bb1ab","Type":"ContainerDied","Data":"0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb"} Oct 30 04:29:43.612481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.612446 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c" event={"ID":"60b4e089-0655-4e15-a46e-a50de07bb1ab","Type":"ContainerDied","Data":"a405871d00eca4e00d4bb2a51d61587c31caea677951747e8ae9c5b56b95f0c2"} Oct 30 04:29:43.619473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.619326 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:43.622746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.622715 4793 scope.go:122] "RemoveContainer" containerID="0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1" Oct 30 04:29:43.623067 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:29:43.623048 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1\": container with ID starting with 0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1 not found: ID does not exist" containerID="0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1" Oct 30 04:29:43.623138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.623077 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1"} err="failed to get container status \"0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1\": rpc error: code = NotFound desc = could not find container \"0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1\": container with ID starting with 0ed4107db175fbb1ed426d9a3e12b5be1077a9e6e99eda0fe682a94e02f10eb1 not found: ID does not exist" Oct 30 04:29:43.623138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.623096 4793 scope.go:122] "RemoveContainer" containerID="0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb" Oct 30 04:29:43.632484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.632463 4793 scope.go:122] "RemoveContainer" containerID="0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb" Oct 30 04:29:43.632720 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:29:43.632702 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb\": container with ID starting with 0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb not found: ID does not exist" containerID="0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb" Oct 30 04:29:43.632775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.632726 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb"} err="failed to get container status \"0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb\": rpc error: code = NotFound desc = could not find container \"0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb\": container with ID starting with 0a45d09fa744f9d1908cdc663182d38b4a0ceeb5be682ae830f79071fc5987cb not found: ID does not exist" Oct 30 04:29:43.684548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.684506 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c"] Oct 30 04:29:43.689283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.689258 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ca7a81-7489fddf54-bcg4c"] Oct 30 04:29:43.707716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.707686 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm"] Oct 30 04:29:43.712680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:43.712655 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ca7a81-6d5dfd6df8-wqmzm"] Oct 30 04:29:44.847642 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:44.847605 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60b4e089-0655-4e15-a46e-a50de07bb1ab" path="/var/lib/kubelet/pods/60b4e089-0655-4e15-a46e-a50de07bb1ab/volumes" Oct 30 04:29:44.848143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:44.848127 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5168c8-44f4-4ef6-84de-1a9437aca7a2" path="/var/lib/kubelet/pods/df5168c8-44f4-4ef6-84de-1a9437aca7a2/volumes" Oct 30 04:29:45.187167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:45.187130 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:45.418704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:45.418648 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:46.636805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.636708 4793 generic.go:356] "Generic (PLEG): container finished" podID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerID="ebc1654ad9e3bf3fd29e6cc29229738b62a125f1e04d021227259041aa16fc84" exitCode=0 Oct 30 04:29:46.636805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.636786 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" event={"ID":"5dc09d10-3490-43e5-9d1f-f52fed67d037","Type":"ContainerDied","Data":"ebc1654ad9e3bf3fd29e6cc29229738b62a125f1e04d021227259041aa16fc84"} Oct 30 04:29:46.637290 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.636850 4793 scope.go:122] "RemoveContainer" containerID="32ece55d2c07dfd9eced0b507e04ad174c042f5d548e7a7a7b8acd293be1ba5a" Oct 30 04:29:46.706211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.705344 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:46.821264 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.821241 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:29:46.838977 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.838955 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-certs\") pod \"5dc09d10-3490-43e5-9d1f-f52fed67d037\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " Oct 30 04:29:46.839092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.838987 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc09d10-3490-43e5-9d1f-f52fed67d037-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc09d10-3490-43e5-9d1f-f52fed67d037-istiod-ca-cert\") pod \"5dc09d10-3490-43e5-9d1f-f52fed67d037\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " Oct 30 04:29:46.839092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839055 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-podinfo\") pod \"5dc09d10-3490-43e5-9d1f-f52fed67d037\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " Oct 30 04:29:46.839222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839091 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-token\" (UniqueName: \"kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-token\") pod \"5dc09d10-3490-43e5-9d1f-f52fed67d037\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " Oct 30 04:29:46.839222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839129 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-envoy\") pod \"5dc09d10-3490-43e5-9d1f-f52fed67d037\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " Oct 30 04:29:46.839222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839213 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-kube-api-access-ttckx\" (UniqueName: \"kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-kube-api-access-ttckx\") pod \"5dc09d10-3490-43e5-9d1f-f52fed67d037\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " Oct 30 04:29:46.839377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839255 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-credential-socket\") pod \"5dc09d10-3490-43e5-9d1f-f52fed67d037\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " Oct 30 04:29:46.839377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839283 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-socket\") pod \"5dc09d10-3490-43e5-9d1f-f52fed67d037\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " Oct 30 04:29:46.839377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839319 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-data\") pod \"5dc09d10-3490-43e5-9d1f-f52fed67d037\" (UID: \"5dc09d10-3490-43e5-9d1f-f52fed67d037\") " Oct 30 04:29:46.839595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839380 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-certs" pod "5dc09d10-3490-43e5-9d1f-f52fed67d037" (UID: "5dc09d10-3490-43e5-9d1f-f52fed67d037"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:46.839595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839437 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc09d10-3490-43e5-9d1f-f52fed67d037-istiod-ca-cert" pod "5dc09d10-3490-43e5-9d1f-f52fed67d037" (UID: "5dc09d10-3490-43e5-9d1f-f52fed67d037"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:29:46.839705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839635 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-credential-socket" pod "5dc09d10-3490-43e5-9d1f-f52fed67d037" (UID: "5dc09d10-3490-43e5-9d1f-f52fed67d037"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:46.839705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839678 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-socket" pod "5dc09d10-3490-43e5-9d1f-f52fed67d037" (UID: "5dc09d10-3490-43e5-9d1f-f52fed67d037"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:46.839922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839847 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:46.839922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839871 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc09d10-3490-43e5-9d1f-f52fed67d037-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:46.839922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839884 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:46.839922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.839900 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:46.840207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.840004 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-data" pod "5dc09d10-3490-43e5-9d1f-f52fed67d037" (UID: "5dc09d10-3490-43e5-9d1f-f52fed67d037"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:46.841896 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.841854 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-token" pod "5dc09d10-3490-43e5-9d1f-f52fed67d037" (UID: "5dc09d10-3490-43e5-9d1f-f52fed67d037"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:46.842028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.841945 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-envoy" pod "5dc09d10-3490-43e5-9d1f-f52fed67d037" (UID: "5dc09d10-3490-43e5-9d1f-f52fed67d037"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:46.842128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.842104 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-podinfo" pod "5dc09d10-3490-43e5-9d1f-f52fed67d037" (UID: "5dc09d10-3490-43e5-9d1f-f52fed67d037"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:29:46.842370 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.842339 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-kube-api-access-ttckx" pod "5dc09d10-3490-43e5-9d1f-f52fed67d037" (UID: "5dc09d10-3490-43e5-9d1f-f52fed67d037"). InnerVolumeSpecName "kube-api-access-ttckx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:46.928069 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.928034 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:46.941120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.941091 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttckx\" (UniqueName: \"kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-kube-api-access-ttckx\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:46.941120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.941122 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:46.941344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.941153 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:46.941344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.941167 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:46.941344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:46.941182 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc09d10-3490-43e5-9d1f-f52fed67d037-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:47.313624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:47.313592 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:47.537629 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:47.537587 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" Oct 30 04:29:47.587489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:47.586800 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:47.647667 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:47.647634 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" event={"ID":"5dc09d10-3490-43e5-9d1f-f52fed67d037","Type":"ContainerDied","Data":"f0f92ac4320054bb3861b7e43dcb3dedb7d076de889cf3e797a27ba45d02100e"} Oct 30 04:29:47.648100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:47.648084 4793 scope.go:122] "RemoveContainer" containerID="ebc1654ad9e3bf3fd29e6cc29229738b62a125f1e04d021227259041aa16fc84" Oct 30 04:29:47.648256 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:47.647651 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc" Oct 30 04:29:47.707549 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:47.707500 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc"] Oct 30 04:29:47.728496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:47.728460 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-baa474-openshift-default-f84445469-wfpwc"] Oct 30 04:29:47.807039 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:47.806997 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw"] Oct 30 04:29:47.807909 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:47.807852 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" containerID="cri-o://64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b" gracePeriod=30 Oct 30 04:29:48.071923 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:48.071893 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:48.544700 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:48.544624 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" Oct 30 04:29:48.822513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:48.822481 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7"] Oct 30 04:29:48.823116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:48.823068 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" podUID="3c50c74d-2900-4872-b08b-882118c38c1d" containerName="maas-api" containerID="cri-o://47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583" gracePeriod=30 Oct 30 04:29:48.865893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:48.865857 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" path="/var/lib/kubelet/pods/5dc09d10-3490-43e5-9d1f-f52fed67d037/volumes" Oct 30 04:29:49.002558 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.002462 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:49.132133 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.132105 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:29:49.192232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.192191 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c50c74d-2900-4872-b08b-882118c38c1d-kube-api-access-hcr88\" (UniqueName: \"kubernetes.io/projected/3c50c74d-2900-4872-b08b-882118c38c1d-kube-api-access-hcr88\") pod \"3c50c74d-2900-4872-b08b-882118c38c1d\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " Oct 30 04:29:49.192662 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.192637 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-maas-api-tls\") pod \"3c50c74d-2900-4872-b08b-882118c38c1d\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " Oct 30 04:29:49.193116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.193094 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-metrics-certs\") pod \"3c50c74d-2900-4872-b08b-882118c38c1d\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " Oct 30 04:29:49.193814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.193787 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c50c74d-2900-4872-b08b-882118c38c1d-openshift-service-ca" pod "3c50c74d-2900-4872-b08b-882118c38c1d" (UID: "3c50c74d-2900-4872-b08b-882118c38c1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:29:49.193911 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.193829 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c50c74d-2900-4872-b08b-882118c38c1d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c50c74d-2900-4872-b08b-882118c38c1d-openshift-service-ca\") pod \"3c50c74d-2900-4872-b08b-882118c38c1d\" (UID: \"3c50c74d-2900-4872-b08b-882118c38c1d\") " Oct 30 04:29:49.194396 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.194377 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c50c74d-2900-4872-b08b-882118c38c1d-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:49.201691 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.201664 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-maas-api-tls" pod "3c50c74d-2900-4872-b08b-882118c38c1d" (UID: "3c50c74d-2900-4872-b08b-882118c38c1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:29:49.201803 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.201789 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c50c74d-2900-4872-b08b-882118c38c1d-kube-api-access-hcr88" pod "3c50c74d-2900-4872-b08b-882118c38c1d" (UID: "3c50c74d-2900-4872-b08b-882118c38c1d"). InnerVolumeSpecName "kube-api-access-hcr88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:49.203717 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.203691 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-metrics-certs" pod "3c50c74d-2900-4872-b08b-882118c38c1d" (UID: "3c50c74d-2900-4872-b08b-882118c38c1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:29:49.216998 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.216966 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:49.301441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.296378 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcr88\" (UniqueName: \"kubernetes.io/projected/3c50c74d-2900-4872-b08b-882118c38c1d-kube-api-access-hcr88\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:49.301441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.296414 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:49.301441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.296471 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50c74d-2900-4872-b08b-882118c38c1d-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:49.627536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.627451 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:49.712460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.694809 4793 generic.go:356] "Generic (PLEG): container finished" podID="3c50c74d-2900-4872-b08b-882118c38c1d" containerID="47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583" exitCode=0 Oct 30 04:29:49.712460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.695079 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" event={"ID":"3c50c74d-2900-4872-b08b-882118c38c1d","Type":"ContainerDied","Data":"47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583"} Oct 30 04:29:49.712460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.695172 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" event={"ID":"3c50c74d-2900-4872-b08b-882118c38c1d","Type":"ContainerDied","Data":"034b8376719f3f4a2dff4162f5e75112413d7022158ebcdc3d57089307aaa46b"} Oct 30 04:29:49.712460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.695214 4793 scope.go:122] "RemoveContainer" containerID="47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583" Oct 30 04:29:49.712460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.695466 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7" Oct 30 04:29:49.743943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.743903 4793 scope.go:122] "RemoveContainer" containerID="47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583" Oct 30 04:29:49.744351 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:29:49.744273 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583\": container with ID starting with 47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583 not found: ID does not exist" containerID="47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583" Oct 30 04:29:49.744351 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.744320 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583"} err="failed to get container status \"47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583\": rpc error: code = NotFound desc = could not find container \"47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583\": container with ID starting with 47128603f32c2ad924fdb7e449bc02ac06d5ca40cf8e2d7e48519f4746742583 not found: ID does not exist" Oct 30 04:29:49.798117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.798077 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7"] Oct 30 04:29:49.810233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.810199 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b1a417-dc686bf4f-z59f7"] Oct 30 04:29:49.870178 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:49.870140 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:50.009682 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.009646 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m"] Oct 30 04:29:50.010609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010573 4793 cpu_manager.go:414] "containerMap: removing container" podUID="df5168c8-44f4-4ef6-84de-1a9437aca7a2" containerName="maas-api" Oct 30 04:29:50.010609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010595 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5168c8-44f4-4ef6-84de-1a9437aca7a2" containerName="maas-api" Oct 30 04:29:50.010804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010639 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3c50c74d-2900-4872-b08b-882118c38c1d" containerName="maas-api" Oct 30 04:29:50.010804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010648 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c50c74d-2900-4872-b08b-882118c38c1d" containerName="maas-api" Oct 30 04:29:50.010804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010670 4793 cpu_manager.go:414] "containerMap: removing container" podUID="60b4e089-0655-4e15-a46e-a50de07bb1ab" containerName="payload-processing" Oct 30 04:29:50.010804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010679 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60b4e089-0655-4e15-a46e-a50de07bb1ab" containerName="payload-processing" Oct 30 04:29:50.010804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010698 4793 cpu_manager.go:414] "containerMap: removing container" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerName="istio-proxy" Oct 30 04:29:50.010804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010706 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerName="istio-proxy" Oct 30 04:29:50.010804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010766 4793 cpu_manager.go:414] "containerMap: removing container" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerName="istio-proxy" Oct 30 04:29:50.010804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010774 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerName="istio-proxy" Oct 30 04:29:50.011207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010893 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c50c74d-2900-4872-b08b-882118c38c1d" containerName="maas-api" Oct 30 04:29:50.011207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010906 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="60b4e089-0655-4e15-a46e-a50de07bb1ab" containerName="payload-processing" Oct 30 04:29:50.011207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010916 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerName="istio-proxy" Oct 30 04:29:50.011207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010930 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc09d10-3490-43e5-9d1f-f52fed67d037" containerName="istio-proxy" Oct 30 04:29:50.011207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.010940 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5168c8-44f4-4ef6-84de-1a9437aca7a2" containerName="maas-api" Oct 30 04:29:50.020457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.020403 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.037872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.037718 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m"] Oct 30 04:29:50.042784 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.042758 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-58cdc8-openshift-default-dockercfg-nr4df\"" Oct 30 04:29:50.120917 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.120877 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-podinfo\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.121106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.120941 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-socket\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.121106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.121045 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istiod-ca-cert\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.121223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.121123 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-credential-socket\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.121223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.121209 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl4mb\" (UniqueName: \"kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-kube-api-access-tl4mb\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.121323 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.121241 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-token\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.121323 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.121274 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-envoy\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.121507 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.121489 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-certs\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.121604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.121587 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-data\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.195143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.195096 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:50.223282 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.223243 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-certs\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.223494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.223332 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-data\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.223494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.223373 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-podinfo\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.223494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.223450 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-socket\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.223494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.223485 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istiod-ca-cert\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.223759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.223532 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-credential-socket\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.223759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.223588 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl4mb\" (UniqueName: \"kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-kube-api-access-tl4mb\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.223759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.223629 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-token\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.223759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.223668 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-envoy\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.223759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.223736 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-certs\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.225068 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.224541 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-socket\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.225068 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.224770 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-data\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.225068 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.224827 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istiod-ca-cert\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.225068 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.225012 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-credential-socket\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.230449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.227726 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-envoy\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.230449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.228245 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-podinfo\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.261052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.260663 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-token\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.278295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.278263 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl4mb\" (UniqueName: \"kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-kube-api-access-tl4mb\") pod \"e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.352951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.352678 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:50.443478 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.436100 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:50.741788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.731873 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:50.799240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.790245 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m"] Oct 30 04:29:50.804091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.804047 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:29:50.804309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.804281 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:29:50.804482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.804461 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:29:50.866110 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.866070 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c50c74d-2900-4872-b08b-882118c38c1d" path="/var/lib/kubelet/pods/3c50c74d-2900-4872-b08b-882118c38c1d/volumes" Oct 30 04:29:50.995074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:50.994938 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:51.676154 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:51.676120 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:51.719979 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:51.719942 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" event={"ID":"a64b9de2-a39e-4d00-8283-1fad481ca2d2","Type":"ContainerStarted","Data":"a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40"} Oct 30 04:29:51.720176 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:51.720008 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" event={"ID":"a64b9de2-a39e-4d00-8283-1fad481ca2d2","Type":"ContainerStarted","Data":"704c912dc1d13e21c9780ea55952e08c9e5892a75825cfeb7323391a23e00017"} Oct 30 04:29:51.799070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:51.799011 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" podStartSLOduration=2.798993637 podStartE2EDuration="2.798993637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:51.796995758 +0000 UTC m=+1915.082870534" watchObservedRunningTime="2026-10-30 04:29:51.798993637 +0000 UTC m=+1915.084868411" Oct 30 04:29:51.910340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:51.910292 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:52.146900 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:52.146822 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:52.357440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:52.356163 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:52.364442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:52.363807 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:52.727728 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:52.727698 4793 generic.go:356] "Generic (PLEG): container finished" podID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerID="42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04" exitCode=0 Oct 30 04:29:52.727932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:52.727770 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" event={"ID":"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf","Type":"ContainerDied","Data":"42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04"} Oct 30 04:29:52.728180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:52.728155 4793 scope.go:122] "RemoveContainer" containerID="42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04" Oct 30 04:29:52.728301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:52.728158 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:52.729118 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:52.729061 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:29:52.729719 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:52.729691 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:29:52.729809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:52.729756 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:29:52.729809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:52.729791 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:29:52.907505 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:52.907313 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:53.114595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.114570 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:29:53.222628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.222599 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-credential-socket\") pod \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " Oct 30 04:29:53.223111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.222643 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-kube-api-access-shh49\" (UniqueName: \"kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-kube-api-access-shh49\") pod \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " Oct 30 04:29:53.223111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.222674 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-podinfo\") pod \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " Oct 30 04:29:53.223111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.222753 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-certs\") pod \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " Oct 30 04:29:53.223111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.222801 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-envoy\") pod \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " Oct 30 04:29:53.223111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.222849 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-data\") pod \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " Oct 30 04:29:53.223111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.222924 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-credential-socket" pod "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" (UID: "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:53.223111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.222952 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istiod-ca-cert\") pod \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " Oct 30 04:29:53.223111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.223018 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-token\" (UniqueName: \"kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-token\") pod \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " Oct 30 04:29:53.223604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.223085 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-socket\") pod \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\" (UID: \"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f\") " Oct 30 04:29:53.223604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.223130 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-certs" pod "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" (UID: "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:53.223604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.223341 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-data" pod "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" (UID: "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:53.223834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.223809 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:53.223916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.223859 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:53.223916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.223876 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:53.224122 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.224103 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-socket" pod "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" (UID: "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:53.224310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.224291 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istiod-ca-cert" pod "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" (UID: "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:29:53.230791 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.230757 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-podinfo" pod "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" (UID: "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:29:53.231011 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.230988 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-kube-api-access-shh49" pod "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" (UID: "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f"). InnerVolumeSpecName "kube-api-access-shh49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:53.232362 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.232337 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-envoy" pod "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" (UID: "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:29:53.232597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.232578 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-token" pod "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" (UID: "ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:29:53.329512 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.328047 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:53.339490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.339109 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:53.339490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.339144 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:53.339490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.339159 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shh49\" (UniqueName: \"kubernetes.io/projected/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-kube-api-access-shh49\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:53.339490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.339176 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:53.339490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.339189 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:53.339490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.339202 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:29:53.718850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.718777 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:53.740594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.740554 4793 generic.go:356] "Generic (PLEG): container finished" podID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerID="64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b" exitCode=0 Oct 30 04:29:53.740773 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.740610 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" Oct 30 04:29:53.740773 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.740637 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" event={"ID":"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f","Type":"ContainerDied","Data":"64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b"} Oct 30 04:29:53.740773 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.740692 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw" event={"ID":"ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f","Type":"ContainerDied","Data":"7da15ccdaadbf9929fac3f20e5aed5451a774fb2192f5b40dcdb3733381b4547"} Oct 30 04:29:53.740773 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.740734 4793 scope.go:122] "RemoveContainer" containerID="64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b" Oct 30 04:29:53.747530 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.747502 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" event={"ID":"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf","Type":"ContainerStarted","Data":"f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336"} Oct 30 04:29:53.757436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.753456 4793 scope.go:122] "RemoveContainer" containerID="0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9" Oct 30 04:29:53.760784 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.760727 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:29:53.761247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.760948 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 04:29:53.761247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.761028 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 04:29:53.789426 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.789342 4793 scope.go:122] "RemoveContainer" containerID="64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b" Oct 30 04:29:53.789730 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:29:53.789701 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b\": container with ID starting with 64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b not found: ID does not exist" containerID="64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b" Oct 30 04:29:53.789778 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.789740 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b"} err="failed to get container status \"64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b\": rpc error: code = NotFound desc = could not find container \"64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b\": container with ID starting with 64a8dfe3e8fb6bc4183f9f10a6ff8e36f03e9765f7a18e429125236c82e0dc7b not found: ID does not exist" Oct 30 04:29:53.789778 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.789756 4793 scope.go:122] "RemoveContainer" containerID="0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9" Oct 30 04:29:53.790088 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:29:53.790049 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9\": container with ID starting with 0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9 not found: ID does not exist" containerID="0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9" Oct 30 04:29:53.790162 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.790087 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9"} err="failed to get container status \"0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9\": rpc error: code = NotFound desc = could not find container \"0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9\": container with ID starting with 0ca7c777253a20782d03412f6df5580a1c370698c58b77613fbc6391bdd16cc9 not found: ID does not exist" Oct 30 04:29:53.814791 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.814753 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw"] Oct 30 04:29:53.825706 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.825676 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ca7a81-openshift-default-8b85498c5-m7hfw"] Oct 30 04:29:53.941829 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:53.941792 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:54.219935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:54.219888 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:54.442943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:54.442904 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:54.765444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:54.760757 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 04:29:54.765444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:54.760816 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 04:29:54.781918 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:54.781880 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:54.846470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:54.846436 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" path="/var/lib/kubelet/pods/ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f/volumes" Oct 30 04:29:55.010373 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.010342 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:55.292486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.292452 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d"] Oct 30 04:29:55.293110 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.293089 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" Oct 30 04:29:55.293222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.293111 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" Oct 30 04:29:55.293312 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.293298 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" Oct 30 04:29:55.293364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.293320 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" Oct 30 04:29:55.296883 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.296860 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.314999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.313507 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-71b161-openshift-default-dockercfg-z72s6\"" Oct 30 04:29:55.342454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.341726 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d"] Oct 30 04:29:55.384442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.383235 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-envoy\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.384442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.383330 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wbt\" (UniqueName: \"kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-kube-api-access-q5wbt\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.384442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.383359 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-credential-socket\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.384442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.383433 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-certs\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.384442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.383489 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62379909-e590-4685-9735-5f3def3f7d99-istiod-ca-cert\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.384442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.383547 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-socket\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.384442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.383593 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62379909-e590-4685-9735-5f3def3f7d99-istio-podinfo\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.384442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.383678 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-istio-token\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.384442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.383709 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-data\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.486455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.486024 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-socket\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.486455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.486084 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62379909-e590-4685-9735-5f3def3f7d99-istio-podinfo\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.486455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.486167 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-istio-token\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.486455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.486196 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-data\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.486455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.486280 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-envoy\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.486455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.486354 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wbt\" (UniqueName: \"kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-kube-api-access-q5wbt\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.486455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.486381 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-credential-socket\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.486455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.486460 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-certs\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.487356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.486511 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62379909-e590-4685-9735-5f3def3f7d99-istiod-ca-cert\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.502479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.487474 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62379909-e590-4685-9735-5f3def3f7d99-istiod-ca-cert\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.502479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.497927 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:55.502479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.498982 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-socket\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.525443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.506135 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-credential-socket\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.525443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.506386 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-certs\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.525443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.515910 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-data\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.550634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.550559 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wbt\" (UniqueName: \"kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-kube-api-access-q5wbt\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.563011 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.562973 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62379909-e590-4685-9735-5f3def3f7d99-istio-podinfo\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.563484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.563455 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-envoy\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.570445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.564596 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-istio-token\") pod \"e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.637199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.637126 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:55.732655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.732626 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:55.789445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.789314 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 04:29:55.789445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.789380 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 04:29:55.986405 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.986367 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4"] Oct 30 04:29:55.987286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.987260 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" Oct 30 04:29:55.987286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.987284 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0f9d7d-88e3-4dd1-a41b-0ced1bcc7b7f" containerName="istio-proxy" Oct 30 04:29:55.987830 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.987797 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:55.990550 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:55.990532 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" Oct 30 04:29:56.002086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.002063 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d"] Oct 30 04:29:56.002777 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:29:56.002755 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62379909_e590_4685_9735_5f3def3f7d99.slice/crio-4514349305b722e14f812266f97f2a65d450843cde88226838e0acab217620ba WatchSource:0}: Error finding container 4514349305b722e14f812266f97f2a65d450843cde88226838e0acab217620ba: Status 404 returned error can't find the container with id 4514349305b722e14f812266f97f2a65d450843cde88226838e0acab217620ba Oct 30 04:29:56.005647 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.005618 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:29:56.005738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.005680 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:29:56.005738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.005708 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:29:56.014497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.014476 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dbabed\"" Oct 30 04:29:56.015933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.015913 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dbabed-dockercfg-vv5vr\"" Oct 30 04:29:56.019898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.019876 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4"] Oct 30 04:29:56.121870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.116336 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4\" (UID: \"8dd09b98-8af9-4092-a06a-4f420e9ba6b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" Oct 30 04:29:56.121870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.116408 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvrp\" (UniqueName: \"kubernetes.io/projected/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-kube-api-access-9kvrp\") pod \"payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4\" (UID: \"8dd09b98-8af9-4092-a06a-4f420e9ba6b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" Oct 30 04:29:56.269496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.269396 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:56.282767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.277363 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4\" (UID: \"8dd09b98-8af9-4092-a06a-4f420e9ba6b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" Oct 30 04:29:56.282977 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.282959 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvrp\" (UniqueName: \"kubernetes.io/projected/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-kube-api-access-9kvrp\") pod \"payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4\" (UID: \"8dd09b98-8af9-4092-a06a-4f420e9ba6b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" Oct 30 04:29:56.283557 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.278203 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4\" (UID: \"8dd09b98-8af9-4092-a06a-4f420e9ba6b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" Oct 30 04:29:56.314890 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.314734 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvrp\" (UniqueName: \"kubernetes.io/projected/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-kube-api-access-9kvrp\") pod \"payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4\" (UID: \"8dd09b98-8af9-4092-a06a-4f420e9ba6b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" Oct 30 04:29:56.495404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.495366 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:56.606873 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.606780 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" Oct 30 04:29:56.770451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.762809 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 04:29:56.770451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.762868 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 04:29:56.808107 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.808074 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" event={"ID":"62379909-e590-4685-9735-5f3def3f7d99","Type":"ContainerStarted","Data":"701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498"} Oct 30 04:29:56.808277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.808131 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" event={"ID":"62379909-e590-4685-9735-5f3def3f7d99","Type":"ContainerStarted","Data":"4514349305b722e14f812266f97f2a65d450843cde88226838e0acab217620ba"} Oct 30 04:29:56.880508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.872325 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" podStartSLOduration=1.872305245 podStartE2EDuration="1.872305245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:56.852718461 +0000 UTC m=+1920.138593248" watchObservedRunningTime="2026-10-30 04:29:56.872305245 +0000 UTC m=+1920.158180023" Oct 30 04:29:57.005660 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:56.990659 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4"] Oct 30 04:29:57.638553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.638523 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:57.645553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.645523 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:57.647001 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.646970 4793 scope.go:122] "RemoveContainer" containerID="dc26c49d90ecf6c099ffa418c228a1009d766fe5c748e9d9444321b622cc0a27" Oct 30 04:29:57.785441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.781519 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 04:29:57.785441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.781580 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 04:29:57.828465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.824534 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" event={"ID":"8dd09b98-8af9-4092-a06a-4f420e9ba6b1","Type":"ContainerStarted","Data":"1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359"} Oct 30 04:29:57.828465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.824591 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" event={"ID":"8dd09b98-8af9-4092-a06a-4f420e9ba6b1","Type":"ContainerStarted","Data":"f4acfc81bb381691ca49261b7b85476f3b17ae2d9e48ca8167eae2b601df4a1e"} Oct 30 04:29:57.828465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.824619 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:57.828465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.824875 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" Oct 30 04:29:57.828465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.826752 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:29:57.828834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.828506 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:57.937090 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.937037 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" podStartSLOduration=2.937023905 podStartE2EDuration="2.937023905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:57.888850077 +0000 UTC m=+1921.174724865" watchObservedRunningTime="2026-10-30 04:29:57.937023905 +0000 UTC m=+1921.222898731" Oct 30 04:29:58.008440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:57.999915 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z"] Oct 30 04:29:58.022134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.020810 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" Oct 30 04:29:58.026057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.020665 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z"] Oct 30 04:29:58.038884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.038076 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-58cdc8\"" Oct 30 04:29:58.038884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.038720 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-58cdc8-dockercfg-vdk94\"" Oct 30 04:29:58.090532 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.090315 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:58.120883 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.120849 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz2r4\" (UniqueName: \"kubernetes.io/projected/23e14557-01b4-4800-980a-0ab377967fcf-kube-api-access-xz2r4\") pod \"payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z\" (UID: \"23e14557-01b4-4800-980a-0ab377967fcf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" Oct 30 04:29:58.121065 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.120933 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23e14557-01b4-4800-980a-0ab377967fcf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z\" (UID: \"23e14557-01b4-4800-980a-0ab377967fcf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" Oct 30 04:29:58.222456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.222351 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz2r4\" (UniqueName: \"kubernetes.io/projected/23e14557-01b4-4800-980a-0ab377967fcf-kube-api-access-xz2r4\") pod \"payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z\" (UID: \"23e14557-01b4-4800-980a-0ab377967fcf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" Oct 30 04:29:58.222640 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.222497 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23e14557-01b4-4800-980a-0ab377967fcf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z\" (UID: \"23e14557-01b4-4800-980a-0ab377967fcf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" Oct 30 04:29:58.223447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.223404 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23e14557-01b4-4800-980a-0ab377967fcf-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z\" (UID: \"23e14557-01b4-4800-980a-0ab377967fcf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" Oct 30 04:29:58.288868 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.288836 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz2r4\" (UniqueName: \"kubernetes.io/projected/23e14557-01b4-4800-980a-0ab377967fcf-kube-api-access-xz2r4\") pod \"payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z\" (UID: \"23e14557-01b4-4800-980a-0ab377967fcf\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" Oct 30 04:29:58.347169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.346970 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" Oct 30 04:29:58.489491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.487991 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:58.779012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.778894 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 04:29:58.779012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.778953 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 04:29:58.914345 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.913133 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:58.915278 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:58.915252 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z"] Oct 30 04:29:58.916591 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:29:58.916563 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23e14557_01b4_4800_980a_0ab377967fcf.slice/crio-52fe65d01111d0600216297ed843f2198a801d21e954fd965b5c4963bd3c5b65 WatchSource:0}: Error finding container 52fe65d01111d0600216297ed843f2198a801d21e954fd965b5c4963bd3c5b65: Status 404 returned error can't find the container with id 52fe65d01111d0600216297ed843f2198a801d21e954fd965b5c4963bd3c5b65 Oct 30 04:29:59.187485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:59.187454 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:29:59.466211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:59.466125 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:59.685400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:59.685364 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:29:59.761211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:59.761146 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:29:59.851685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:59.851647 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" event={"ID":"23e14557-01b4-4800-980a-0ab377967fcf","Type":"ContainerStarted","Data":"b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8"} Oct 30 04:29:59.852092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:59.851731 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" event={"ID":"23e14557-01b4-4800-980a-0ab377967fcf","Type":"ContainerStarted","Data":"52fe65d01111d0600216297ed843f2198a801d21e954fd965b5c4963bd3c5b65"} Oct 30 04:29:59.853040 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:59.853015 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:29:59.896189 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:29:59.896138 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" podStartSLOduration=2.896125087 podStartE2EDuration="2.896125087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:29:59.892525065 +0000 UTC m=+1923.178399859" watchObservedRunningTime="2026-10-30 04:29:59.896125087 +0000 UTC m=+1923.181999865" Oct 30 04:30:00.173025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.172933 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt"] Oct 30 04:30:00.176679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.176656 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" Oct 30 04:30:00.178344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.178309 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96"] Oct 30 04:30:00.182957 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.182932 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" Oct 30 04:30:00.186614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.186589 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d01373a-7c83-4fc3-878d-538513b31b06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt\" (UID: \"4d01373a-7c83-4fc3-878d-538513b31b06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" Oct 30 04:30:00.187030 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.187009 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6zrx\" (UniqueName: \"kubernetes.io/projected/4d01373a-7c83-4fc3-878d-538513b31b06-kube-api-access-s6zrx\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt\" (UID: \"4d01373a-7c83-4fc3-878d-538513b31b06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" Oct 30 04:30:00.209827 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.209801 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8"] Oct 30 04:30:00.218198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.218164 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt"] Oct 30 04:30:00.218319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.218299 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" Oct 30 04:30:00.220555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.220517 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96"] Oct 30 04:30:00.225672 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.225653 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw"] Oct 30 04:30:00.229364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.229347 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" Oct 30 04:30:00.229817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.229792 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8"] Oct 30 04:30:00.242008 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.241974 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw"] Oct 30 04:30:00.277776 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.277748 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:30:00.288505 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.288477 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6zrx\" (UniqueName: \"kubernetes.io/projected/4d01373a-7c83-4fc3-878d-538513b31b06-kube-api-access-s6zrx\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt\" (UID: \"4d01373a-7c83-4fc3-878d-538513b31b06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" Oct 30 04:30:00.288620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.288535 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed72997-93d4-45a2-a9c8-084b259814a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw\" (UID: \"0ed72997-93d4-45a2-a9c8-084b259814a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" Oct 30 04:30:00.288620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.288601 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d01373a-7c83-4fc3-878d-538513b31b06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt\" (UID: \"4d01373a-7c83-4fc3-878d-538513b31b06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" Oct 30 04:30:00.288807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.288646 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4b5t\" (UniqueName: \"kubernetes.io/projected/6087f4a1-d193-4a0d-9c37-2c91a733de4f-kube-api-access-p4b5t\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96\" (UID: \"6087f4a1-d193-4a0d-9c37-2c91a733de4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" Oct 30 04:30:00.288961 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.288935 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6087f4a1-d193-4a0d-9c37-2c91a733de4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96\" (UID: \"6087f4a1-d193-4a0d-9c37-2c91a733de4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" Oct 30 04:30:00.289093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.288976 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqmsq\" (UniqueName: \"kubernetes.io/projected/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-kube-api-access-dqmsq\") pod \"maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8\" (UID: \"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" Oct 30 04:30:00.289093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.289001 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svhsr\" (UniqueName: \"kubernetes.io/projected/0ed72997-93d4-45a2-a9c8-084b259814a0-kube-api-access-svhsr\") pod \"maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw\" (UID: \"0ed72997-93d4-45a2-a9c8-084b259814a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" Oct 30 04:30:00.289202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.289118 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8\" (UID: \"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" Oct 30 04:30:00.289545 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.289526 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d01373a-7c83-4fc3-878d-538513b31b06-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt\" (UID: \"4d01373a-7c83-4fc3-878d-538513b31b06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" Oct 30 04:30:00.312218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.312196 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6zrx\" (UniqueName: \"kubernetes.io/projected/4d01373a-7c83-4fc3-878d-538513b31b06-kube-api-access-s6zrx\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt\" (UID: \"4d01373a-7c83-4fc3-878d-538513b31b06\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" Oct 30 04:30:00.390741 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.390711 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed72997-93d4-45a2-a9c8-084b259814a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw\" (UID: \"0ed72997-93d4-45a2-a9c8-084b259814a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" Oct 30 04:30:00.390920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.390804 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4b5t\" (UniqueName: \"kubernetes.io/projected/6087f4a1-d193-4a0d-9c37-2c91a733de4f-kube-api-access-p4b5t\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96\" (UID: \"6087f4a1-d193-4a0d-9c37-2c91a733de4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" Oct 30 04:30:00.390920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.390841 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6087f4a1-d193-4a0d-9c37-2c91a733de4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96\" (UID: \"6087f4a1-d193-4a0d-9c37-2c91a733de4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" Oct 30 04:30:00.390920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.390864 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqmsq\" (UniqueName: \"kubernetes.io/projected/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-kube-api-access-dqmsq\") pod \"maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8\" (UID: \"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" Oct 30 04:30:00.390920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.390894 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svhsr\" (UniqueName: \"kubernetes.io/projected/0ed72997-93d4-45a2-a9c8-084b259814a0-kube-api-access-svhsr\") pod \"maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw\" (UID: \"0ed72997-93d4-45a2-a9c8-084b259814a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" Oct 30 04:30:00.391135 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.390996 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8\" (UID: \"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" Oct 30 04:30:00.391519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.391494 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed72997-93d4-45a2-a9c8-084b259814a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw\" (UID: \"0ed72997-93d4-45a2-a9c8-084b259814a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" Oct 30 04:30:00.391628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.391587 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6087f4a1-d193-4a0d-9c37-2c91a733de4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96\" (UID: \"6087f4a1-d193-4a0d-9c37-2c91a733de4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" Oct 30 04:30:00.391732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.391714 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8\" (UID: \"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" Oct 30 04:30:00.414614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.414590 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4b5t\" (UniqueName: \"kubernetes.io/projected/6087f4a1-d193-4a0d-9c37-2c91a733de4f-kube-api-access-p4b5t\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96\" (UID: \"6087f4a1-d193-4a0d-9c37-2c91a733de4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" Oct 30 04:30:00.414792 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.414773 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqmsq\" (UniqueName: \"kubernetes.io/projected/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-kube-api-access-dqmsq\") pod \"maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8\" (UID: \"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" Oct 30 04:30:00.414874 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.414852 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svhsr\" (UniqueName: \"kubernetes.io/projected/0ed72997-93d4-45a2-a9c8-084b259814a0-kube-api-access-svhsr\") pod \"maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw\" (UID: \"0ed72997-93d4-45a2-a9c8-084b259814a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" Oct 30 04:30:00.513255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.513230 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" Oct 30 04:30:00.527733 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.527710 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" Oct 30 04:30:00.537979 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.537959 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" Oct 30 04:30:00.549153 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.549125 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" Oct 30 04:30:00.826631 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:30:00.826599 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d01373a_7c83_4fc3_878d_538513b31b06.slice/crio-5f06c2374235b303a2c2675033c95a10ffd484cbcc1e514fb5979e53661d6004 WatchSource:0}: Error finding container 5f06c2374235b303a2c2675033c95a10ffd484cbcc1e514fb5979e53661d6004: Status 404 returned error can't find the container with id 5f06c2374235b303a2c2675033c95a10ffd484cbcc1e514fb5979e53661d6004 Oct 30 04:30:00.827369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.827341 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt"] Oct 30 04:30:00.865678 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:30:00.865647 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6087f4a1_d193_4a0d_9c37_2c91a733de4f.slice/crio-8918c66e97c4e35f81884e087161a36c29b9639e0fd239eb5b79bc672ec2960e WatchSource:0}: Error finding container 8918c66e97c4e35f81884e087161a36c29b9639e0fd239eb5b79bc672ec2960e: Status 404 returned error can't find the container with id 8918c66e97c4e35f81884e087161a36c29b9639e0fd239eb5b79bc672ec2960e Oct 30 04:30:00.866674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.866644 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" event={"ID":"4d01373a-7c83-4fc3-878d-538513b31b06","Type":"ContainerStarted","Data":"5f06c2374235b303a2c2675033c95a10ffd484cbcc1e514fb5979e53661d6004"} Oct 30 04:30:00.867407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.867387 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" Oct 30 04:30:00.871108 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.871083 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96"] Oct 30 04:30:00.883838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.883812 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8"] Oct 30 04:30:00.912013 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:30:00.911989 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed72997_93d4_45a2_a9c8_084b259814a0.slice/crio-2178272f9ef3d2e49ed27ea1f0811dbfebb8568db5501a2b727109b5473d4bc6 WatchSource:0}: Error finding container 2178272f9ef3d2e49ed27ea1f0811dbfebb8568db5501a2b727109b5473d4bc6: Status 404 returned error can't find the container with id 2178272f9ef3d2e49ed27ea1f0811dbfebb8568db5501a2b727109b5473d4bc6 Oct 30 04:30:00.912473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:00.912456 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw"] Oct 30 04:30:01.066134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:01.066103 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:01.331646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:01.308565 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:01.880872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:01.880840 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" event={"ID":"4d01373a-7c83-4fc3-878d-538513b31b06","Type":"ContainerStarted","Data":"e2e66d0852b21c74a5b4d57c2557e3beda90f6fe986635bb0abbacd789867f38"} Oct 30 04:30:01.882864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:01.882840 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" event={"ID":"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58","Type":"ContainerStarted","Data":"829eddab6a1c03442b76ca0f25d2b06efce53923758d6121e737333c4171cd44"} Oct 30 04:30:01.882957 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:01.882888 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" event={"ID":"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58","Type":"ContainerStarted","Data":"6b53df472b888dbbedaa2d0819e854b41ffb48f062d0ef508933f487cf34c895"} Oct 30 04:30:01.884672 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:01.884643 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" event={"ID":"0ed72997-93d4-45a2-a9c8-084b259814a0","Type":"ContainerStarted","Data":"1854379a38c6f673b4cf28386e861cf2dc9ff7e581b1bb9e326b358561edc5ca"} Oct 30 04:30:01.884803 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:01.884688 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" event={"ID":"0ed72997-93d4-45a2-a9c8-084b259814a0","Type":"ContainerStarted","Data":"2178272f9ef3d2e49ed27ea1f0811dbfebb8568db5501a2b727109b5473d4bc6"} Oct 30 04:30:01.886222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:01.886199 4793 generic.go:356] "Generic (PLEG): container finished" podID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" containerID="ee5d5c99d83323ec5f874ef347004233d9036be3ce6302c561697c596f9aecb5" exitCode=60 Oct 30 04:30:01.886344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:01.886321 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" event={"ID":"6087f4a1-d193-4a0d-9c37-2c91a733de4f","Type":"ContainerDied","Data":"ee5d5c99d83323ec5f874ef347004233d9036be3ce6302c561697c596f9aecb5"} Oct 30 04:30:01.886449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:01.886370 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" event={"ID":"6087f4a1-d193-4a0d-9c37-2c91a733de4f","Type":"ContainerStarted","Data":"8918c66e97c4e35f81884e087161a36c29b9639e0fd239eb5b79bc672ec2960e"} Oct 30 04:30:01.886508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:01.886452 4793 scope.go:122] "RemoveContainer" containerID="ee5d5c99d83323ec5f874ef347004233d9036be3ce6302c561697c596f9aecb5" Oct 30 04:30:01.923275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:01.923227 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" podStartSLOduration=1.923213421 podStartE2EDuration="1.923213421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:01.920391302 +0000 UTC m=+1925.206266079" watchObservedRunningTime="2026-10-30 04:30:01.923213421 +0000 UTC m=+1925.209088197" Oct 30 04:30:02.011481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.011407 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" podStartSLOduration=2.011393732 podStartE2EDuration="2.011393732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:02.011350715 +0000 UTC m=+1925.297225490" watchObservedRunningTime="2026-10-30 04:30:02.011393732 +0000 UTC m=+1925.297268507" Oct 30 04:30:02.061794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.061754 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" podStartSLOduration=2.061741937 podStartE2EDuration="2.061741937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:02.058928781 +0000 UTC m=+1925.344803580" watchObservedRunningTime="2026-10-30 04:30:02.061741937 +0000 UTC m=+1925.347616740" Oct 30 04:30:02.905187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.905151 4793 generic.go:356] "Generic (PLEG): container finished" podID="4d01373a-7c83-4fc3-878d-538513b31b06" containerID="e2e66d0852b21c74a5b4d57c2557e3beda90f6fe986635bb0abbacd789867f38" exitCode=60 Oct 30 04:30:02.905649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.905241 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" event={"ID":"4d01373a-7c83-4fc3-878d-538513b31b06","Type":"ContainerDied","Data":"e2e66d0852b21c74a5b4d57c2557e3beda90f6fe986635bb0abbacd789867f38"} Oct 30 04:30:02.905748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.905734 4793 scope.go:122] "RemoveContainer" containerID="e2e66d0852b21c74a5b4d57c2557e3beda90f6fe986635bb0abbacd789867f38" Oct 30 04:30:02.911612 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.910071 4793 generic.go:356] "Generic (PLEG): container finished" podID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" containerID="829eddab6a1c03442b76ca0f25d2b06efce53923758d6121e737333c4171cd44" exitCode=60 Oct 30 04:30:02.911612 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.910250 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" event={"ID":"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58","Type":"ContainerDied","Data":"829eddab6a1c03442b76ca0f25d2b06efce53923758d6121e737333c4171cd44"} Oct 30 04:30:02.911612 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.910485 4793 scope.go:122] "RemoveContainer" containerID="829eddab6a1c03442b76ca0f25d2b06efce53923758d6121e737333c4171cd44" Oct 30 04:30:02.913187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.913150 4793 generic.go:356] "Generic (PLEG): container finished" podID="0ed72997-93d4-45a2-a9c8-084b259814a0" containerID="1854379a38c6f673b4cf28386e861cf2dc9ff7e581b1bb9e326b358561edc5ca" exitCode=60 Oct 30 04:30:02.913384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.913210 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" event={"ID":"0ed72997-93d4-45a2-a9c8-084b259814a0","Type":"ContainerDied","Data":"1854379a38c6f673b4cf28386e861cf2dc9ff7e581b1bb9e326b358561edc5ca"} Oct 30 04:30:02.913477 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.913467 4793 scope.go:122] "RemoveContainer" containerID="1854379a38c6f673b4cf28386e861cf2dc9ff7e581b1bb9e326b358561edc5ca" Oct 30 04:30:02.915386 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.915363 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" event={"ID":"6087f4a1-d193-4a0d-9c37-2c91a733de4f","Type":"ContainerStarted","Data":"fa8a0f80c07f4f54287b3f2906f94374b00c19288d88fb0241c77dd992a4d30c"} Oct 30 04:30:02.915610 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.915593 4793 scope.go:122] "RemoveContainer" containerID="fa8a0f80c07f4f54287b3f2906f94374b00c19288d88fb0241c77dd992a4d30c" Oct 30 04:30:02.915815 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:02.915796 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96_odh-ai-gateway-infra(6087f4a1-d193-4a0d-9c37-2c91a733de4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" Oct 30 04:30:02.935978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:02.935878 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:03.923496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:03.923373 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" event={"ID":"0ed72997-93d4-45a2-a9c8-084b259814a0","Type":"ContainerStarted","Data":"a0740b0b8a4137d91e8a515b7a459636579ed46da29c72b69443355d63060bda"} Oct 30 04:30:03.925456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:03.925428 4793 generic.go:356] "Generic (PLEG): container finished" podID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" containerID="fa8a0f80c07f4f54287b3f2906f94374b00c19288d88fb0241c77dd992a4d30c" exitCode=60 Oct 30 04:30:03.925601 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:03.925508 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" event={"ID":"6087f4a1-d193-4a0d-9c37-2c91a733de4f","Type":"ContainerDied","Data":"fa8a0f80c07f4f54287b3f2906f94374b00c19288d88fb0241c77dd992a4d30c"} Oct 30 04:30:03.925601 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:03.925559 4793 scope.go:122] "RemoveContainer" containerID="ee5d5c99d83323ec5f874ef347004233d9036be3ce6302c561697c596f9aecb5" Oct 30 04:30:03.925728 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:03.925710 4793 scope.go:122] "RemoveContainer" containerID="fa8a0f80c07f4f54287b3f2906f94374b00c19288d88fb0241c77dd992a4d30c" Oct 30 04:30:03.926008 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:03.925982 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96_odh-ai-gateway-infra(6087f4a1-d193-4a0d-9c37-2c91a733de4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" Oct 30 04:30:03.927641 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:03.927622 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" event={"ID":"4d01373a-7c83-4fc3-878d-538513b31b06","Type":"ContainerStarted","Data":"d3fc0f1771fac1b6421dc3a65b1ae9eda35eb7d8befc531ff1e612945efdc14e"} Oct 30 04:30:03.929292 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:03.929269 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" event={"ID":"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58","Type":"ContainerStarted","Data":"abe57b941814302ea13e9f2d2d8d5e7ad4133ce8b331dce2c9e9fdfb2e272caa"} Oct 30 04:30:04.768407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.768371 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:04.939002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.938973 4793 scope.go:122] "RemoveContainer" containerID="fa8a0f80c07f4f54287b3f2906f94374b00c19288d88fb0241c77dd992a4d30c" Oct 30 04:30:04.939475 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:04.939296 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96_odh-ai-gateway-infra(6087f4a1-d193-4a0d-9c37-2c91a733de4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" Oct 30 04:30:04.940716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.940688 4793 generic.go:356] "Generic (PLEG): container finished" podID="4d01373a-7c83-4fc3-878d-538513b31b06" containerID="d3fc0f1771fac1b6421dc3a65b1ae9eda35eb7d8befc531ff1e612945efdc14e" exitCode=60 Oct 30 04:30:04.940852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.940774 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" event={"ID":"4d01373a-7c83-4fc3-878d-538513b31b06","Type":"ContainerDied","Data":"d3fc0f1771fac1b6421dc3a65b1ae9eda35eb7d8befc531ff1e612945efdc14e"} Oct 30 04:30:04.940852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.940832 4793 scope.go:122] "RemoveContainer" containerID="e2e66d0852b21c74a5b4d57c2557e3beda90f6fe986635bb0abbacd789867f38" Oct 30 04:30:04.941028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.941006 4793 scope.go:122] "RemoveContainer" containerID="d3fc0f1771fac1b6421dc3a65b1ae9eda35eb7d8befc531ff1e612945efdc14e" Oct 30 04:30:04.941299 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:04.941278 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt_odh-ai-gateway-infra(4d01373a-7c83-4fc3-878d-538513b31b06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" Oct 30 04:30:04.942841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.942822 4793 generic.go:356] "Generic (PLEG): container finished" podID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" containerID="abe57b941814302ea13e9f2d2d8d5e7ad4133ce8b331dce2c9e9fdfb2e272caa" exitCode=60 Oct 30 04:30:04.942932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.942905 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" event={"ID":"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58","Type":"ContainerDied","Data":"abe57b941814302ea13e9f2d2d8d5e7ad4133ce8b331dce2c9e9fdfb2e272caa"} Oct 30 04:30:04.943094 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.943079 4793 scope.go:122] "RemoveContainer" containerID="abe57b941814302ea13e9f2d2d8d5e7ad4133ce8b331dce2c9e9fdfb2e272caa" Oct 30 04:30:04.943344 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:04.943311 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8_odh-ai-gateway-infra(dddb9cf4-e7e5-4581-8ae4-9f80345b7e58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" Oct 30 04:30:04.944797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.944776 4793 generic.go:356] "Generic (PLEG): container finished" podID="0ed72997-93d4-45a2-a9c8-084b259814a0" containerID="a0740b0b8a4137d91e8a515b7a459636579ed46da29c72b69443355d63060bda" exitCode=60 Oct 30 04:30:04.944888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.944838 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" event={"ID":"0ed72997-93d4-45a2-a9c8-084b259814a0","Type":"ContainerDied","Data":"a0740b0b8a4137d91e8a515b7a459636579ed46da29c72b69443355d63060bda"} Oct 30 04:30:04.945035 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.945021 4793 scope.go:122] "RemoveContainer" containerID="a0740b0b8a4137d91e8a515b7a459636579ed46da29c72b69443355d63060bda" Oct 30 04:30:04.945245 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:04.945229 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw_odh-ai-gateway-infra(0ed72997-93d4-45a2-a9c8-084b259814a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" Oct 30 04:30:04.953630 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.953580 4793 scope.go:122] "RemoveContainer" containerID="829eddab6a1c03442b76ca0f25d2b06efce53923758d6121e737333c4171cd44" Oct 30 04:30:04.973066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:04.973045 4793 scope.go:122] "RemoveContainer" containerID="1854379a38c6f673b4cf28386e861cf2dc9ff7e581b1bb9e326b358561edc5ca" Oct 30 04:30:05.953725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:05.953683 4793 scope.go:122] "RemoveContainer" containerID="a0740b0b8a4137d91e8a515b7a459636579ed46da29c72b69443355d63060bda" Oct 30 04:30:05.954172 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:05.954021 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw_odh-ai-gateway-infra(0ed72997-93d4-45a2-a9c8-084b259814a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" Oct 30 04:30:05.955681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:05.955657 4793 scope.go:122] "RemoveContainer" containerID="d3fc0f1771fac1b6421dc3a65b1ae9eda35eb7d8befc531ff1e612945efdc14e" Oct 30 04:30:05.955966 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:05.955943 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt_odh-ai-gateway-infra(4d01373a-7c83-4fc3-878d-538513b31b06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" Oct 30 04:30:05.957350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:05.957331 4793 scope.go:122] "RemoveContainer" containerID="abe57b941814302ea13e9f2d2d8d5e7ad4133ce8b331dce2c9e9fdfb2e272caa" Oct 30 04:30:05.957575 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:05.957548 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8_odh-ai-gateway-infra(dddb9cf4-e7e5-4581-8ae4-9f80345b7e58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" Oct 30 04:30:06.064689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:06.064653 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:06.325015 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:06.315411 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:07.208433 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:07.208386 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:08.243947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:08.243912 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:08.463750 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:08.463713 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:08.836865 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:08.836832 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" Oct 30 04:30:09.034159 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:09.034116 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:09.269003 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:09.268944 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:09.328136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:09.328096 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w"] Oct 30 04:30:09.328379 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:09.328347 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" containerID="cri-o://f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563" gracePeriod=30 Oct 30 04:30:09.524570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:09.512497 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:10.951788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:10.951729 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:11.166741 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:11.166704 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:11.219529 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:11.219366 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:30:11.889649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:11.889624 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" Oct 30 04:30:12.394533 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.394494 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:12.582307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.582273 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw"] Oct 30 04:30:12.587822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.587805 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.606790 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.606762 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw"] Oct 30 04:30:12.607521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.607497 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-58cdc8-serving-cert\"" Oct 30 04:30:12.697153 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.697111 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-metrics-certs\") pod \"maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.697313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.697238 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-maas-api-tls\") pod \"maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.697379 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.697364 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qftvk\" (UniqueName: \"kubernetes.io/projected/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-kube-api-access-qftvk\") pod \"maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.697502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.697486 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.798570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.798535 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-metrics-certs\") pod \"maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.798747 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.798627 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-maas-api-tls\") pod \"maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.798822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.798758 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qftvk\" (UniqueName: \"kubernetes.io/projected/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-kube-api-access-qftvk\") pod \"maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.798910 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.798869 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.799726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.799694 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.801250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.801227 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-metrics-certs\") pod \"maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.801390 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.801369 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-maas-api-tls\") pod \"maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.834439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.834403 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qftvk\" (UniqueName: \"kubernetes.io/projected/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-kube-api-access-qftvk\") pod \"maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:12.859585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.859553 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:12.902157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:12.902123 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:13.097223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:13.097189 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:13.328866 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:30:13.328834 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc2c258_ffe4_4e16_bf5e_996d41b90f45.slice/crio-c210664a9481b504111508a0d189fff3f8dbcc047d43b6e90508c30c578f26a4 WatchSource:0}: Error finding container c210664a9481b504111508a0d189fff3f8dbcc047d43b6e90508c30c578f26a4: Status 404 returned error can't find the container with id c210664a9481b504111508a0d189fff3f8dbcc047d43b6e90508c30c578f26a4 Oct 30 04:30:13.329050 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:13.329027 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw"] Oct 30 04:30:13.838927 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:13.838904 4793 scope.go:122] "RemoveContainer" containerID="fa8a0f80c07f4f54287b3f2906f94374b00c19288d88fb0241c77dd992a4d30c" Oct 30 04:30:14.038098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.038068 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" event={"ID":"ccc2c258-ffe4-4e16-bf5e-996d41b90f45","Type":"ContainerStarted","Data":"5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d"} Oct 30 04:30:14.038309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.038125 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" event={"ID":"ccc2c258-ffe4-4e16-bf5e-996d41b90f45","Type":"ContainerStarted","Data":"c210664a9481b504111508a0d189fff3f8dbcc047d43b6e90508c30c578f26a4"} Oct 30 04:30:14.038309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.038295 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:14.084130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.084027 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" podStartSLOduration=1.81018131 podStartE2EDuration="2.084013456s" totalImagesPullingTime="273.832146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:14.080504178 +0000 UTC m=+1937.366378954" watchObservedRunningTime="2026-10-30 04:30:14.084013456 +0000 UTC m=+1937.369888230" Oct 30 04:30:14.629737 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.629710 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:30:14.744198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.744120 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-credential-socket\") pod \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " Oct 30 04:30:14.744361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.744236 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-data\") pod \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " Oct 30 04:30:14.744361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.744287 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-socket\") pod \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " Oct 30 04:30:14.744361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.744345 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istiod-ca-cert\") pod \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " Oct 30 04:30:14.744580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.744380 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-token\" (UniqueName: \"kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-token\") pod \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " Oct 30 04:30:14.744580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.744410 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-podinfo\") pod \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " Oct 30 04:30:14.744580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.744498 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-kube-api-access-qdnq7\" (UniqueName: \"kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-kube-api-access-qdnq7\") pod \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " Oct 30 04:30:14.744580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.744539 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-data" pod "7fc4700a-fc65-4c54-ab00-66df2dd352c5" (UID: "7fc4700a-fc65-4c54-ab00-66df2dd352c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:30:14.744795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.744605 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-socket" pod "7fc4700a-fc65-4c54-ab00-66df2dd352c5" (UID: "7fc4700a-fc65-4c54-ab00-66df2dd352c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:30:14.744987 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.744967 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-certs\") pod \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " Oct 30 04:30:14.745085 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.745033 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-envoy\") pod \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\" (UID: \"7fc4700a-fc65-4c54-ab00-66df2dd352c5\") " Oct 30 04:30:14.745543 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.745516 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:14.745543 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.745539 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:14.745705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.745629 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-credential-socket" pod "7fc4700a-fc65-4c54-ab00-66df2dd352c5" (UID: "7fc4700a-fc65-4c54-ab00-66df2dd352c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:30:14.745874 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.745852 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-certs" pod "7fc4700a-fc65-4c54-ab00-66df2dd352c5" (UID: "7fc4700a-fc65-4c54-ab00-66df2dd352c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:30:14.746025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.746005 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istiod-ca-cert" pod "7fc4700a-fc65-4c54-ab00-66df2dd352c5" (UID: "7fc4700a-fc65-4c54-ab00-66df2dd352c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:30:14.747592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.747565 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-podinfo" pod "7fc4700a-fc65-4c54-ab00-66df2dd352c5" (UID: "7fc4700a-fc65-4c54-ab00-66df2dd352c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:30:14.747735 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.747697 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-kube-api-access-qdnq7" pod "7fc4700a-fc65-4c54-ab00-66df2dd352c5" (UID: "7fc4700a-fc65-4c54-ab00-66df2dd352c5"). InnerVolumeSpecName "kube-api-access-qdnq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:30:14.749063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.749035 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-envoy" pod "7fc4700a-fc65-4c54-ab00-66df2dd352c5" (UID: "7fc4700a-fc65-4c54-ab00-66df2dd352c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:30:14.749626 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.749596 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-token" pod "7fc4700a-fc65-4c54-ab00-66df2dd352c5" (UID: "7fc4700a-fc65-4c54-ab00-66df2dd352c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:30:14.756305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.755880 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:14.838202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.838178 4793 scope.go:122] "RemoveContainer" containerID="a0740b0b8a4137d91e8a515b7a459636579ed46da29c72b69443355d63060bda" Oct 30 04:30:14.846748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.846711 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:14.846748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.846739 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:14.846748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.846748 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc4700a-fc65-4c54-ab00-66df2dd352c5-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:14.847242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.846761 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:14.847242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.846774 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:14.847242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.846784 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc4700a-fc65-4c54-ab00-66df2dd352c5-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:14.847242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:14.846797 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdnq7\" (UniqueName: \"kubernetes.io/projected/7fc4700a-fc65-4c54-ab00-66df2dd352c5-kube-api-access-qdnq7\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:15.048073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.047994 4793 generic.go:356] "Generic (PLEG): container finished" podID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" containerID="00711245fdaabc937ef74f84efc1f2eb9d5b745c2ad0d151ca18e64dab9f8ada" exitCode=60 Oct 30 04:30:15.048225 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.048067 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" event={"ID":"6087f4a1-d193-4a0d-9c37-2c91a733de4f","Type":"ContainerDied","Data":"00711245fdaabc937ef74f84efc1f2eb9d5b745c2ad0d151ca18e64dab9f8ada"} Oct 30 04:30:15.048225 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.048123 4793 scope.go:122] "RemoveContainer" containerID="fa8a0f80c07f4f54287b3f2906f94374b00c19288d88fb0241c77dd992a4d30c" Oct 30 04:30:15.048612 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.048585 4793 scope.go:122] "RemoveContainer" containerID="00711245fdaabc937ef74f84efc1f2eb9d5b745c2ad0d151ca18e64dab9f8ada" Oct 30 04:30:15.048919 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:15.048893 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96_odh-ai-gateway-infra(6087f4a1-d193-4a0d-9c37-2c91a733de4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" Oct 30 04:30:15.050144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.050120 4793 generic.go:356] "Generic (PLEG): container finished" podID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerID="f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563" exitCode=0 Oct 30 04:30:15.050243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.050168 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" Oct 30 04:30:15.050243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.050206 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" event={"ID":"7fc4700a-fc65-4c54-ab00-66df2dd352c5","Type":"ContainerDied","Data":"f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563"} Oct 30 04:30:15.050323 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.050244 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w" event={"ID":"7fc4700a-fc65-4c54-ab00-66df2dd352c5","Type":"ContainerDied","Data":"264771e9fa015111b3edbd08b705442a342ccdac628415a4bce657b9372b255c"} Oct 30 04:30:15.075030 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.075013 4793 scope.go:122] "RemoveContainer" containerID="f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563" Oct 30 04:30:15.084428 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.084404 4793 scope.go:122] "RemoveContainer" containerID="dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f" Oct 30 04:30:15.093276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.093251 4793 scope.go:122] "RemoveContainer" containerID="f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563" Oct 30 04:30:15.093535 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:15.093516 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563\": container with ID starting with f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563 not found: ID does not exist" containerID="f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563" Oct 30 04:30:15.093598 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.093541 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563"} err="failed to get container status \"f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563\": rpc error: code = NotFound desc = could not find container \"f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563\": container with ID starting with f039d83c34dbd60af2ae66d15f2d8766566ffee299ab8d88d3c03cb5eee7e563 not found: ID does not exist" Oct 30 04:30:15.093598 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.093559 4793 scope.go:122] "RemoveContainer" containerID="dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f" Oct 30 04:30:15.093842 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:15.093822 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f\": container with ID starting with dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f not found: ID does not exist" containerID="dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f" Oct 30 04:30:15.093902 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.093849 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f"} err="failed to get container status \"dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f\": rpc error: code = NotFound desc = could not find container \"dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f\": container with ID starting with dee29f6e3b1d268e5866f9ced976c2e66d1ceeefcd837e80d7c9c3aa34e7883f not found: ID does not exist" Oct 30 04:30:15.158951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.158921 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w"] Oct 30 04:30:15.166366 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.166335 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-763bde-openshift-default-54bfd857c9-vrb4w"] Oct 30 04:30:15.838488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:15.838462 4793 scope.go:122] "RemoveContainer" containerID="d3fc0f1771fac1b6421dc3a65b1ae9eda35eb7d8befc531ff1e612945efdc14e" Oct 30 04:30:16.029750 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.029719 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph"] Oct 30 04:30:16.033315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.033287 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe861886-8ffph"] Oct 30 04:30:16.059666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.059642 4793 generic.go:356] "Generic (PLEG): container finished" podID="0ed72997-93d4-45a2-a9c8-084b259814a0" containerID="7176baaf9d207d4ace6a30cb9bc51e7da1aefed56d049102f38abe7830d8d5ba" exitCode=60 Oct 30 04:30:16.059788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.059712 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" event={"ID":"0ed72997-93d4-45a2-a9c8-084b259814a0","Type":"ContainerDied","Data":"7176baaf9d207d4ace6a30cb9bc51e7da1aefed56d049102f38abe7830d8d5ba"} Oct 30 04:30:16.059788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.059766 4793 scope.go:122] "RemoveContainer" containerID="a0740b0b8a4137d91e8a515b7a459636579ed46da29c72b69443355d63060bda" Oct 30 04:30:16.060157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.060129 4793 scope.go:122] "RemoveContainer" containerID="7176baaf9d207d4ace6a30cb9bc51e7da1aefed56d049102f38abe7830d8d5ba" Oct 30 04:30:16.060505 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:16.060475 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw_odh-ai-gateway-infra(0ed72997-93d4-45a2-a9c8-084b259814a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" Oct 30 04:30:16.125516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.125486 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96"] Oct 30 04:30:16.309052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.309022 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" Oct 30 04:30:16.374284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.374254 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6087f4a1-d193-4a0d-9c37-2c91a733de4f-kube-api-access-p4b5t\" (UniqueName: \"kubernetes.io/projected/6087f4a1-d193-4a0d-9c37-2c91a733de4f-kube-api-access-p4b5t\") pod \"6087f4a1-d193-4a0d-9c37-2c91a733de4f\" (UID: \"6087f4a1-d193-4a0d-9c37-2c91a733de4f\") " Oct 30 04:30:16.374462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.374360 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6087f4a1-d193-4a0d-9c37-2c91a733de4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6087f4a1-d193-4a0d-9c37-2c91a733de4f-maas-api-service-ca\") pod \"6087f4a1-d193-4a0d-9c37-2c91a733de4f\" (UID: \"6087f4a1-d193-4a0d-9c37-2c91a733de4f\") " Oct 30 04:30:16.375121 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.375093 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6087f4a1-d193-4a0d-9c37-2c91a733de4f-maas-api-service-ca" pod "6087f4a1-d193-4a0d-9c37-2c91a733de4f" (UID: "6087f4a1-d193-4a0d-9c37-2c91a733de4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:30:16.384457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.376608 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6087f4a1-d193-4a0d-9c37-2c91a733de4f-kube-api-access-p4b5t" pod "6087f4a1-d193-4a0d-9c37-2c91a733de4f" (UID: "6087f4a1-d193-4a0d-9c37-2c91a733de4f"). InnerVolumeSpecName "kube-api-access-p4b5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:30:16.476392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.476363 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4b5t\" (UniqueName: \"kubernetes.io/projected/6087f4a1-d193-4a0d-9c37-2c91a733de4f-kube-api-access-p4b5t\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:16.476392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.476389 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6087f4a1-d193-4a0d-9c37-2c91a733de4f-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:16.847309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.847278 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" path="/var/lib/kubelet/pods/7fc4700a-fc65-4c54-ab00-66df2dd352c5/volumes" Oct 30 04:30:16.847767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.847751 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0189496-3809-4080-b12b-42152c45e7b1" path="/var/lib/kubelet/pods/a0189496-3809-4080-b12b-42152c45e7b1/volumes" Oct 30 04:30:16.909529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:16.909497 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:17.076780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.076753 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" Oct 30 04:30:17.077257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.076794 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96" event={"ID":"6087f4a1-d193-4a0d-9c37-2c91a733de4f","Type":"ContainerDied","Data":"8918c66e97c4e35f81884e087161a36c29b9639e0fd239eb5b79bc672ec2960e"} Oct 30 04:30:17.077257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.076860 4793 scope.go:122] "RemoveContainer" containerID="00711245fdaabc937ef74f84efc1f2eb9d5b745c2ad0d151ca18e64dab9f8ada" Oct 30 04:30:17.079472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.079447 4793 generic.go:356] "Generic (PLEG): container finished" podID="4d01373a-7c83-4fc3-878d-538513b31b06" containerID="396da39d95a6bea99fd10311d67fe3d2137c0f2e1987b422c5cc510f1ef3ed16" exitCode=60 Oct 30 04:30:17.079671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.079566 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" event={"ID":"4d01373a-7c83-4fc3-878d-538513b31b06","Type":"ContainerDied","Data":"396da39d95a6bea99fd10311d67fe3d2137c0f2e1987b422c5cc510f1ef3ed16"} Oct 30 04:30:17.080171 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.080151 4793 scope.go:122] "RemoveContainer" containerID="396da39d95a6bea99fd10311d67fe3d2137c0f2e1987b422c5cc510f1ef3ed16" Oct 30 04:30:17.080518 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:17.080488 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt_odh-ai-gateway-infra(4d01373a-7c83-4fc3-878d-538513b31b06)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" Oct 30 04:30:17.089532 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.089510 4793 scope.go:122] "RemoveContainer" containerID="d3fc0f1771fac1b6421dc3a65b1ae9eda35eb7d8befc531ff1e612945efdc14e" Oct 30 04:30:17.112874 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.112848 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw"] Oct 30 04:30:17.157441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.152526 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96"] Oct 30 04:30:17.191385 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.190081 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:17.214137 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.200537 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888910-mfh96"] Oct 30 04:30:17.388264 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.388239 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" Oct 30 04:30:17.439125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.439084 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:17.540573 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.540536 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed72997-93d4-45a2-a9c8-084b259814a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed72997-93d4-45a2-a9c8-084b259814a0-maas-api-service-ca\") pod \"0ed72997-93d4-45a2-a9c8-084b259814a0\" (UID: \"0ed72997-93d4-45a2-a9c8-084b259814a0\") " Oct 30 04:30:17.540757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.540659 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed72997-93d4-45a2-a9c8-084b259814a0-kube-api-access-svhsr\" (UniqueName: \"kubernetes.io/projected/0ed72997-93d4-45a2-a9c8-084b259814a0-kube-api-access-svhsr\") pod \"0ed72997-93d4-45a2-a9c8-084b259814a0\" (UID: \"0ed72997-93d4-45a2-a9c8-084b259814a0\") " Oct 30 04:30:17.540967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.540939 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed72997-93d4-45a2-a9c8-084b259814a0-maas-api-service-ca" pod "0ed72997-93d4-45a2-a9c8-084b259814a0" (UID: "0ed72997-93d4-45a2-a9c8-084b259814a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:30:17.541247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.541217 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed72997-93d4-45a2-a9c8-084b259814a0-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:17.542692 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.542665 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed72997-93d4-45a2-a9c8-084b259814a0-kube-api-access-svhsr" pod "0ed72997-93d4-45a2-a9c8-084b259814a0" (UID: "0ed72997-93d4-45a2-a9c8-084b259814a0"). InnerVolumeSpecName "kube-api-access-svhsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:30:17.642779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:17.642697 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svhsr\" (UniqueName: \"kubernetes.io/projected/0ed72997-93d4-45a2-a9c8-084b259814a0-kube-api-access-svhsr\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:18.092603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.092560 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" event={"ID":"0ed72997-93d4-45a2-a9c8-084b259814a0","Type":"ContainerDied","Data":"2178272f9ef3d2e49ed27ea1f0811dbfebb8568db5501a2b727109b5473d4bc6"} Oct 30 04:30:18.093097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.092629 4793 scope.go:122] "RemoveContainer" containerID="7176baaf9d207d4ace6a30cb9bc51e7da1aefed56d049102f38abe7830d8d5ba" Oct 30 04:30:18.093097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.092576 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw" Oct 30 04:30:18.150048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.150021 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw"] Oct 30 04:30:18.153985 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.153959 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3f5508-29888910-rv8rw"] Oct 30 04:30:18.227997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.227971 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt"] Oct 30 04:30:18.423237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.423211 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" Oct 30 04:30:18.553143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.553113 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d01373a-7c83-4fc3-878d-538513b31b06-kube-api-access-s6zrx\" (UniqueName: \"kubernetes.io/projected/4d01373a-7c83-4fc3-878d-538513b31b06-kube-api-access-s6zrx\") pod \"4d01373a-7c83-4fc3-878d-538513b31b06\" (UID: \"4d01373a-7c83-4fc3-878d-538513b31b06\") " Oct 30 04:30:18.553331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.553317 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d01373a-7c83-4fc3-878d-538513b31b06-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d01373a-7c83-4fc3-878d-538513b31b06-maas-api-service-ca\") pod \"4d01373a-7c83-4fc3-878d-538513b31b06\" (UID: \"4d01373a-7c83-4fc3-878d-538513b31b06\") " Oct 30 04:30:18.553750 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.553720 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d01373a-7c83-4fc3-878d-538513b31b06-maas-api-service-ca" pod "4d01373a-7c83-4fc3-878d-538513b31b06" (UID: "4d01373a-7c83-4fc3-878d-538513b31b06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:30:18.553896 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.553804 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d01373a-7c83-4fc3-878d-538513b31b06-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:18.555333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.555303 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d01373a-7c83-4fc3-878d-538513b31b06-kube-api-access-s6zrx" pod "4d01373a-7c83-4fc3-878d-538513b31b06" (UID: "4d01373a-7c83-4fc3-878d-538513b31b06"). InnerVolumeSpecName "kube-api-access-s6zrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:30:18.608726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.608697 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:18.655496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.655413 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6zrx\" (UniqueName: \"kubernetes.io/projected/4d01373a-7c83-4fc3-878d-538513b31b06-kube-api-access-s6zrx\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:18.812440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.811108 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct"] Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.823815 4793 cpu_manager.go:414] "containerMap: removing container" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.823849 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.823870 4793 cpu_manager.go:414] "containerMap: removing container" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.823877 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.823893 4793 cpu_manager.go:414] "containerMap: removing container" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.823900 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.823918 4793 cpu_manager.go:414] "containerMap: removing container" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.823925 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.823936 4793 cpu_manager.go:414] "containerMap: removing container" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.823943 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.823970 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.823977 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824014 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824023 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824043 4793 cpu_manager.go:414] "containerMap: removing container" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824050 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824066 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824073 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824238 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824251 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824261 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824272 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824284 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824294 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824307 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" containerName="cleanup" Oct 30 04:30:18.831441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.824321 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" containerName="cleanup" Oct 30 04:30:18.850487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.846772 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct"] Oct 30 04:30:18.850487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.847641 4793 scope.go:122] "RemoveContainer" containerID="abe57b941814302ea13e9f2d2d8d5e7ad4133ce8b331dce2c9e9fdfb2e272caa" Oct 30 04:30:18.850487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.849214 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" Oct 30 04:30:18.867221 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.865435 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-153a44\"" Oct 30 04:30:18.867221 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.866063 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-153a44-dockercfg-5kb2b\"" Oct 30 04:30:18.874823 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.874791 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:18.885770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.885745 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed72997-93d4-45a2-a9c8-084b259814a0" path="/var/lib/kubelet/pods/0ed72997-93d4-45a2-a9c8-084b259814a0/volumes" Oct 30 04:30:18.886672 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.886649 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" path="/var/lib/kubelet/pods/6087f4a1-d193-4a0d-9c37-2c91a733de4f/volumes" Oct 30 04:30:18.928683 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.928657 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh"] Oct 30 04:30:18.929277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.929262 4793 cpu_manager.go:414] "containerMap: removing container" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" containerName="cleanup" Oct 30 04:30:18.929327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.929279 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" containerName="cleanup" Oct 30 04:30:18.929327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.929294 4793 cpu_manager.go:414] "containerMap: removing container" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" containerName="cleanup" Oct 30 04:30:18.929327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.929299 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" containerName="cleanup" Oct 30 04:30:18.929458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.929378 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="6087f4a1-d193-4a0d-9c37-2c91a733de4f" containerName="cleanup" Oct 30 04:30:18.929458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.929390 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc4700a-fc65-4c54-ab00-66df2dd352c5" containerName="istio-proxy" Oct 30 04:30:18.929458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.929398 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" containerName="cleanup" Oct 30 04:30:18.933033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.933012 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:18.934015 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.933993 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:18.934109 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.934037 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56htq\" (UniqueName: \"kubernetes.io/projected/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-kube-api-access-56htq\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:18.934185 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.934152 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e08ad2-cda0-4914-b839-081b2f791a82-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-153a44-969747556-xr6ct\" (UID: \"73e08ad2-cda0-4914-b839-081b2f791a82\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" Oct 30 04:30:18.934335 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.934301 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8bwr\" (UniqueName: \"kubernetes.io/projected/73e08ad2-cda0-4914-b839-081b2f791a82-kube-api-access-g8bwr\") pod \"payload-processing-e2e-praxis-a-153a44-969747556-xr6ct\" (UID: \"73e08ad2-cda0-4914-b839-081b2f791a82\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" Oct 30 04:30:18.934534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.934514 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-metrics-certs\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:18.934675 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.934654 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:18.953805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.953453 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh"] Oct 30 04:30:18.965532 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:18.965025 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-71b161-serving-cert\"" Oct 30 04:30:19.035500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.035470 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8bwr\" (UniqueName: \"kubernetes.io/projected/73e08ad2-cda0-4914-b839-081b2f791a82-kube-api-access-g8bwr\") pod \"payload-processing-e2e-praxis-a-153a44-969747556-xr6ct\" (UID: \"73e08ad2-cda0-4914-b839-081b2f791a82\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" Oct 30 04:30:19.035650 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.035545 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-metrics-certs\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:19.035650 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.035569 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:19.035746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.035659 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:19.035746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.035690 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56htq\" (UniqueName: \"kubernetes.io/projected/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-kube-api-access-56htq\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:19.035746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.035736 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e08ad2-cda0-4914-b839-081b2f791a82-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-153a44-969747556-xr6ct\" (UID: \"73e08ad2-cda0-4914-b839-081b2f791a82\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" Oct 30 04:30:19.035939 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:19.035785 4793 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-serving-cert: secret "maas-api-e2e-praxis-a-71b161-serving-cert" not found Oct 30 04:30:19.035939 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:19.035848 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-maas-api-tls podName:ea8a64a8-e2ff-4427-85b7-8d559346a8a8 nodeName:}" failed. No retries permitted until 2026-10-30 04:30:19.535827754 +0000 UTC m=+1942.821702526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-maas-api-tls") pod "maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" (UID: "ea8a64a8-e2ff-4427-85b7-8d559346a8a8") : secret "maas-api-e2e-praxis-a-71b161-serving-cert" not found Oct 30 04:30:19.036377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.036357 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:19.036455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.036399 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e08ad2-cda0-4914-b839-081b2f791a82-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-153a44-969747556-xr6ct\" (UID: \"73e08ad2-cda0-4914-b839-081b2f791a82\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" Oct 30 04:30:19.037841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.037823 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-metrics-certs\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:19.060210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.060187 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56htq\" (UniqueName: \"kubernetes.io/projected/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-kube-api-access-56htq\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:19.060584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.060563 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8bwr\" (UniqueName: \"kubernetes.io/projected/73e08ad2-cda0-4914-b839-081b2f791a82-kube-api-access-g8bwr\") pod \"payload-processing-e2e-praxis-a-153a44-969747556-xr6ct\" (UID: \"73e08ad2-cda0-4914-b839-081b2f791a82\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" Oct 30 04:30:19.100966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.100947 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" Oct 30 04:30:19.101356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.100989 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt" event={"ID":"4d01373a-7c83-4fc3-878d-538513b31b06","Type":"ContainerDied","Data":"5f06c2374235b303a2c2675033c95a10ffd484cbcc1e514fb5979e53661d6004"} Oct 30 04:30:19.101356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.101044 4793 scope.go:122] "RemoveContainer" containerID="396da39d95a6bea99fd10311d67fe3d2137c0f2e1987b422c5cc510f1ef3ed16" Oct 30 04:30:19.127164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.127139 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:19.154874 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.154851 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt"] Oct 30 04:30:19.161510 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.161489 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888910-rfxzt"] Oct 30 04:30:19.201823 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.201795 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" Oct 30 04:30:19.346658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.346622 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:19.521807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.521773 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct"] Oct 30 04:30:19.522052 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:30:19.522028 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e08ad2_cda0_4914_b839_081b2f791a82.slice/crio-996214d39ac0af72e1883348985992dd33ac7875f1f79583768ed211441b46f3 WatchSource:0}: Error finding container 996214d39ac0af72e1883348985992dd33ac7875f1f79583768ed211441b46f3: Status 404 returned error can't find the container with id 996214d39ac0af72e1883348985992dd33ac7875f1f79583768ed211441b46f3 Oct 30 04:30:19.541684 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.541655 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:19.544047 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.543971 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:19.546665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.546636 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:19.878170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:19.878142 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh"] Oct 30 04:30:19.878570 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:30:19.878543 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8a64a8_e2ff_4427_85b7_8d559346a8a8.slice/crio-e945bea7a5dedbeaed592a1112d04dbab32a09d56915616bd495069bb697fe20 WatchSource:0}: Error finding container e945bea7a5dedbeaed592a1112d04dbab32a09d56915616bd495069bb697fe20: Status 404 returned error can't find the container with id e945bea7a5dedbeaed592a1112d04dbab32a09d56915616bd495069bb697fe20 Oct 30 04:30:20.056369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:20.055797 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:30:20.168249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:20.168131 4793 generic.go:356] "Generic (PLEG): container finished" podID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" containerID="ba8727f02b87a8c8d50a7b1652006eb5bfa898604c9c23bce1d513d3c275d9e8" exitCode=60 Oct 30 04:30:20.168797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:20.168251 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" event={"ID":"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58","Type":"ContainerDied","Data":"ba8727f02b87a8c8d50a7b1652006eb5bfa898604c9c23bce1d513d3c275d9e8"} Oct 30 04:30:20.168797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:20.168324 4793 scope.go:122] "RemoveContainer" containerID="abe57b941814302ea13e9f2d2d8d5e7ad4133ce8b331dce2c9e9fdfb2e272caa" Oct 30 04:30:20.169673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:20.168994 4793 scope.go:122] "RemoveContainer" containerID="ba8727f02b87a8c8d50a7b1652006eb5bfa898604c9c23bce1d513d3c275d9e8" Oct 30 04:30:20.169673 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:20.169475 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8_odh-ai-gateway-infra(dddb9cf4-e7e5-4581-8ae4-9f80345b7e58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" Oct 30 04:30:20.172912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:20.172885 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" event={"ID":"73e08ad2-cda0-4914-b839-081b2f791a82","Type":"ContainerStarted","Data":"82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2"} Oct 30 04:30:20.173098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:20.172937 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" event={"ID":"73e08ad2-cda0-4914-b839-081b2f791a82","Type":"ContainerStarted","Data":"996214d39ac0af72e1883348985992dd33ac7875f1f79583768ed211441b46f3"} Oct 30 04:30:20.173098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:20.173085 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" Oct 30 04:30:20.176891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:20.176870 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" event={"ID":"ea8a64a8-e2ff-4427-85b7-8d559346a8a8","Type":"ContainerStarted","Data":"e945bea7a5dedbeaed592a1112d04dbab32a09d56915616bd495069bb697fe20"} Oct 30 04:30:20.253391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:20.253347 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" podStartSLOduration=2.253334412 podStartE2EDuration="2.253334412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:20.251665633 +0000 UTC m=+1943.537540411" watchObservedRunningTime="2026-10-30 04:30:20.253334412 +0000 UTC m=+1943.539209188" Oct 30 04:30:20.850148 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:20.850110 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d01373a-7c83-4fc3-878d-538513b31b06" path="/var/lib/kubelet/pods/4d01373a-7c83-4fc3-878d-538513b31b06/volumes" Oct 30 04:30:20.910702 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:20.910673 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:21.190489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:21.190451 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" event={"ID":"ea8a64a8-e2ff-4427-85b7-8d559346a8a8","Type":"ContainerStarted","Data":"16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205"} Oct 30 04:30:21.223893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:21.223861 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8"] Oct 30 04:30:21.237388 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:21.237341 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" podStartSLOduration=2.873325848 podStartE2EDuration="3.237327078s" totalImagesPullingTime="364.00123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:21.234262341 +0000 UTC m=+1944.520137119" watchObservedRunningTime="2026-10-30 04:30:21.237327078 +0000 UTC m=+1944.523201904" Oct 30 04:30:21.415804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:21.415776 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" Oct 30 04:30:21.479297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:21.479218 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-maas-api-service-ca\") pod \"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58\" (UID: \"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58\") " Oct 30 04:30:21.479505 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:21.479300 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-kube-api-access-dqmsq\" (UniqueName: \"kubernetes.io/projected/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-kube-api-access-dqmsq\") pod \"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58\" (UID: \"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58\") " Oct 30 04:30:21.479621 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:21.479597 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-maas-api-service-ca" pod "dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" (UID: "dddb9cf4-e7e5-4581-8ae4-9f80345b7e58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:30:21.479872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:21.479854 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:21.481473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:21.481439 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-kube-api-access-dqmsq" pod "dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" (UID: "dddb9cf4-e7e5-4581-8ae4-9f80345b7e58"). InnerVolumeSpecName "kube-api-access-dqmsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:30:21.580946 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:21.580916 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqmsq\" (UniqueName: \"kubernetes.io/projected/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58-kube-api-access-dqmsq\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:22.198696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:22.198668 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" Oct 30 04:30:22.199127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:22.198701 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8" event={"ID":"dddb9cf4-e7e5-4581-8ae4-9f80345b7e58","Type":"ContainerDied","Data":"6b53df472b888dbbedaa2d0819e854b41ffb48f062d0ef508933f487cf34c895"} Oct 30 04:30:22.199127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:22.198768 4793 scope.go:122] "RemoveContainer" containerID="ba8727f02b87a8c8d50a7b1652006eb5bfa898604c9c23bce1d513d3c275d9e8" Oct 30 04:30:22.199127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:22.199099 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:22.254200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:22.254176 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8"] Oct 30 04:30:22.266470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:22.266446 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f7a36d-29888910-tkqz8"] Oct 30 04:30:22.623851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:22.623745 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:22.846746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:22.846714 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" path="/var/lib/kubelet/pods/dddb9cf4-e7e5-4581-8ae4-9f80345b7e58/volumes" Oct 30 04:30:24.029339 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:24.029303 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:24.417120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:24.417030 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:24.639005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:24.638969 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:24.951845 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:24.951808 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:26.030097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:26.030065 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th"] Oct 30 04:30:26.042622 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:26.042591 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9736ddd-hx5th"] Oct 30 04:30:26.311645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:26.311558 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:26.847521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:26.847491 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36778c6c-6fb6-4ce9-9ca5-0bda78427209" path="/var/lib/kubelet/pods/36778c6c-6fb6-4ce9-9ca5-0bda78427209/volumes" Oct 30 04:30:27.927360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:27.927327 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:28.155379 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:28.155324 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:28.214014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:28.213939 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:30:28.746910 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:28.746872 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:29.638987 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.638939 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:29.695642 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.695612 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg"] Oct 30 04:30:29.696228 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.696213 4793 cpu_manager.go:414] "containerMap: removing container" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" containerName="cleanup" Oct 30 04:30:29.696281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.696229 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" containerName="cleanup" Oct 30 04:30:29.696281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.696247 4793 cpu_manager.go:414] "containerMap: removing container" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" containerName="cleanup" Oct 30 04:30:29.696281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.696253 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" containerName="cleanup" Oct 30 04:30:29.696281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.696280 4793 cpu_manager.go:414] "containerMap: removing container" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" containerName="cleanup" Oct 30 04:30:29.696403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.696286 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" containerName="cleanup" Oct 30 04:30:29.696403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.696395 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" containerName="cleanup" Oct 30 04:30:29.696503 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.696403 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" containerName="cleanup" Oct 30 04:30:29.701682 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.701662 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" Oct 30 04:30:29.718959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.718925 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6a777c-dockercfg-9jz9c\"" Oct 30 04:30:29.720967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.720939 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg"] Oct 30 04:30:29.721648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.721627 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6a777c\"" Oct 30 04:30:29.819441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.818749 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xhh7\" (UniqueName: \"kubernetes.io/projected/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-kube-api-access-6xhh7\") pod \"payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg\" (UID: \"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" Oct 30 04:30:29.819441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.819045 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg\" (UID: \"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" Oct 30 04:30:29.896012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.895869 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:29.920442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.920396 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg\" (UID: \"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" Oct 30 04:30:29.920616 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.920515 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xhh7\" (UniqueName: \"kubernetes.io/projected/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-kube-api-access-6xhh7\") pod \"payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg\" (UID: \"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" Oct 30 04:30:29.921235 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.921212 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg\" (UID: \"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" Oct 30 04:30:29.944074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:29.944053 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xhh7\" (UniqueName: \"kubernetes.io/projected/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-kube-api-access-6xhh7\") pod \"payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg\" (UID: \"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" Oct 30 04:30:30.016192 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:30.016163 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" Oct 30 04:30:30.333820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:30.333791 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg"] Oct 30 04:30:30.523944 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:30.523906 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:31.193618 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:31.193588 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" Oct 30 04:30:31.275211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:31.275176 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" event={"ID":"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9","Type":"ContainerStarted","Data":"24de24043639f1e9b03045d8ba21fa0fbf7d5cc0a57e728275f90a14e0e2e462"} Oct 30 04:30:31.275373 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:31.275230 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" event={"ID":"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9","Type":"ContainerStarted","Data":"73b8d604f97f6a67be15b51be9fd5dbe88e01658be5845fd040de0be3c2f4737"} Oct 30 04:30:31.275451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:31.275367 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" Oct 30 04:30:31.322356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:31.322300 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" podStartSLOduration=2.322286691 podStartE2EDuration="2.322286691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:31.318444068 +0000 UTC m=+1954.604318837" watchObservedRunningTime="2026-10-30 04:30:31.322286691 +0000 UTC m=+1954.608161465" Oct 30 04:30:32.030123 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:32.030090 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr"] Oct 30 04:30:32.042466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:32.042440 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c899505-trvbr"] Oct 30 04:30:32.369805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:32.369717 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:32.846937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:32.846902 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="379aab4d-7a33-4ca0-bbd7-8ffb091638d5" path="/var/lib/kubelet/pods/379aab4d-7a33-4ca0-bbd7-8ffb091638d5/volumes" Oct 30 04:30:34.191853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:34.191820 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:36.004950 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:36.004912 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:38.290140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:38.290089 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:38.501768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:38.501730 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:39.420624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:39.420579 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4"] Oct 30 04:30:39.634703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:39.634666 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:39.911716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:39.897368 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:40.269730 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:40.269692 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:40.511095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:40.511060 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:41.240809 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:41.240622 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:30:41.585060 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:41.584970 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:42.045229 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.045183 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7"] Oct 30 04:30:42.047276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.047248 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddb9cf4-e7e5-4581-8ae4-9f80345b7e58" containerName="cleanup" Oct 30 04:30:42.095563 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.076749 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.095563 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.095263 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-eb9d23-openshift-default-dockercfg-ftwnr\"" Oct 30 04:30:42.101714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.101683 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7"] Oct 30 04:30:42.176443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.176197 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-socket\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.176443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.176265 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-credential-socket\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.176443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.176314 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-podinfo\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.176758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.176510 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg46w\" (UniqueName: \"kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-kube-api-access-jg46w\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.176758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.176552 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istiod-ca-cert\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.176758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.176581 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-token\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.176758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.176615 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-envoy\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.176758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.176668 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-data\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.176758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.176736 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-certs\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.205355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.205306 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:42.300499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.278326 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-credential-socket\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.300845 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.300813 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-podinfo\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.301195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.301173 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg46w\" (UniqueName: \"kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-kube-api-access-jg46w\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.301348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.301331 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istiod-ca-cert\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.301497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.301478 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-token\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.301628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.301610 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-envoy\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.301781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.301766 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-data\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.301943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.301928 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-certs\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.302069 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.302056 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-socket\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.330439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.302617 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-socket\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.330695 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.278876 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-credential-socket\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.330815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.307642 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istiod-ca-cert\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.330922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.307901 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-data\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.331102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.308948 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-certs\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.331217 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.323684 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" Oct 30 04:30:42.331457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.325165 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-podinfo\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.337190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.337158 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-envoy\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.339084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.339056 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg46w\" (UniqueName: \"kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-kube-api-access-jg46w\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.353064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.353029 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-token\") pod \"e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.419622 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.419374 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:42.548033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.547995 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:42.797502 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:30:42.797465 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7805bda5_d2a2_4d8c_87a9_306ffa87fa4a.slice/crio-c22b90fe24b8725370a4dad1b625f3784877307e1604f9102bac790a55a5b5de WatchSource:0}: Error finding container c22b90fe24b8725370a4dad1b625f3784877307e1604f9102bac790a55a5b5de: Status 404 returned error can't find the container with id c22b90fe24b8725370a4dad1b625f3784877307e1604f9102bac790a55a5b5de Oct 30 04:30:42.797928 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.797812 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7"] Oct 30 04:30:42.800347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.800312 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:30:42.800469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.800396 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:30:42.800469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:42.800441 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:30:43.486440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:43.485650 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" event={"ID":"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a","Type":"ContainerStarted","Data":"7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50"} Oct 30 04:30:43.486440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:43.485718 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" event={"ID":"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a","Type":"ContainerStarted","Data":"c22b90fe24b8725370a4dad1b625f3784877307e1604f9102bac790a55a5b5de"} Oct 30 04:30:43.577444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:43.575438 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" podStartSLOduration=2.575399344 podStartE2EDuration="2.575399344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:43.545780708 +0000 UTC m=+1966.831655486" watchObservedRunningTime="2026-10-30 04:30:43.575399344 +0000 UTC m=+1966.861274119" Oct 30 04:30:44.130646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.130608 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:44.420007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.419937 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:44.421533 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.421487 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" podUID="8dd09b98-8af9-4092-a06a-4f420e9ba6b1" containerName="payload-pre-processing" containerID="cri-o://1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359" gracePeriod=25 Oct 30 04:30:44.427918 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.427677 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:44.463562 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:44.463534 4793 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd09b98_8af9_4092_a06a_4f420e9ba6b1.slice/crio-1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359.scope\": RecentStats: unable to find data in memory cache]" Oct 30 04:30:44.501458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.501411 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:44.503952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.503928 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:30:44.756822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.756786 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" Oct 30 04:30:44.841886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.841561 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-plugins-config-volume\") pod \"8dd09b98-8af9-4092-a06a-4f420e9ba6b1\" (UID: \"8dd09b98-8af9-4092-a06a-4f420e9ba6b1\") " Oct 30 04:30:44.841886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.841807 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-kube-api-access-9kvrp\" (UniqueName: \"kubernetes.io/projected/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-kube-api-access-9kvrp\") pod \"8dd09b98-8af9-4092-a06a-4f420e9ba6b1\" (UID: \"8dd09b98-8af9-4092-a06a-4f420e9ba6b1\") " Oct 30 04:30:44.843135 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.842940 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-plugins-config-volume" pod "8dd09b98-8af9-4092-a06a-4f420e9ba6b1" (UID: "8dd09b98-8af9-4092-a06a-4f420e9ba6b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:30:44.871495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.852887 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-kube-api-access-9kvrp" pod "8dd09b98-8af9-4092-a06a-4f420e9ba6b1" (UID: "8dd09b98-8af9-4092-a06a-4f420e9ba6b1"). InnerVolumeSpecName "kube-api-access-9kvrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:30:44.943643 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.943595 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kvrp\" (UniqueName: \"kubernetes.io/projected/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-kube-api-access-9kvrp\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:44.943643 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:44.943635 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dd09b98-8af9-4092-a06a-4f420e9ba6b1-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:30:45.217689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:45.217645 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:45.513869 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:45.513770 4793 generic.go:356] "Generic (PLEG): container finished" podID="8dd09b98-8af9-4092-a06a-4f420e9ba6b1" containerID="1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359" exitCode=0 Oct 30 04:30:45.514321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:45.514293 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" Oct 30 04:30:45.514725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:45.514702 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" event={"ID":"8dd09b98-8af9-4092-a06a-4f420e9ba6b1","Type":"ContainerDied","Data":"1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359"} Oct 30 04:30:45.514834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:45.514753 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4" event={"ID":"8dd09b98-8af9-4092-a06a-4f420e9ba6b1","Type":"ContainerDied","Data":"f4acfc81bb381691ca49261b7b85476f3b17ae2d9e48ca8167eae2b601df4a1e"} Oct 30 04:30:45.514834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:45.514779 4793 scope.go:122] "RemoveContainer" containerID="1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359" Oct 30 04:30:45.527548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:45.527234 4793 scope.go:122] "RemoveContainer" containerID="1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359" Oct 30 04:30:45.527636 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:30:45.527550 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359\": container with ID starting with 1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359 not found: ID does not exist" containerID="1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359" Oct 30 04:30:45.527636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:45.527582 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359"} err="failed to get container status \"1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359\": rpc error: code = NotFound desc = could not find container \"1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359\": container with ID starting with 1cd49147db9be67b19b2a81c99d18e0a6f584ac920ede68a4ae8d3318708f359 not found: ID does not exist" Oct 30 04:30:45.581952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:45.581825 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4"] Oct 30 04:30:45.593620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:45.593586 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dbabed-854d595775-9fmt4"] Oct 30 04:30:46.027971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.027932 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:46.267134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.267090 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:46.484437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.483985 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:46.701579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.701510 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:46.811957 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.811874 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp"] Oct 30 04:30:46.812649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.812619 4793 cpu_manager.go:414] "containerMap: removing container" podUID="8dd09b98-8af9-4092-a06a-4f420e9ba6b1" containerName="payload-pre-processing" Oct 30 04:30:46.812649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.812641 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd09b98-8af9-4092-a06a-4f420e9ba6b1" containerName="payload-pre-processing" Oct 30 04:30:46.812853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.812799 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd09b98-8af9-4092-a06a-4f420e9ba6b1" containerName="payload-pre-processing" Oct 30 04:30:46.815216 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.815197 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:46.832122 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.832063 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-05f827-openshift-default-dockercfg-8rtc4\"" Oct 30 04:30:46.851674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.851631 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd09b98-8af9-4092-a06a-4f420e9ba6b1" path="/var/lib/kubelet/pods/8dd09b98-8af9-4092-a06a-4f420e9ba6b1/volumes" Oct 30 04:30:46.851962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.851945 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp"] Oct 30 04:30:46.877485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.877445 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-certs\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:46.877668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.877500 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-socket\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:46.877668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.877599 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpvdj\" (UniqueName: \"kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-kube-api-access-fpvdj\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:46.877668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.877654 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-credential-socket\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:46.877852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.877738 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-podinfo\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:46.877852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.877811 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-token\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:46.877852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.877841 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31e3220-e28e-432d-9f9a-86ce56e66a78-istiod-ca-cert\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:46.878032 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.878005 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-envoy\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:46.878139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:46.878078 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-data\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.003339 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.003300 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:47.015439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.012827 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-podinfo\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.015439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.012925 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-token\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.015439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.012970 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31e3220-e28e-432d-9f9a-86ce56e66a78-istiod-ca-cert\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.015439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.013330 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-envoy\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.058453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.031448 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-data\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.058453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.031578 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-certs\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.058453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.031623 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-socket\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.058453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.031702 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpvdj\" (UniqueName: \"kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-kube-api-access-fpvdj\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.058453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.031768 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-credential-socket\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.058453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.032157 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-credential-socket\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.058453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.032470 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-data\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.058453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.032695 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-certs\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.058453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.032907 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-socket\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.058453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.048486 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31e3220-e28e-432d-9f9a-86ce56e66a78-istiod-ca-cert\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.066640 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.066429 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpvdj\" (UniqueName: \"kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-kube-api-access-fpvdj\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.069375 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.069326 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-token\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.081280 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.081241 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-podinfo\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.081810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.081766 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-envoy\") pod \"e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.134433 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.132050 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:47.234287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.232868 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:47.470250 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:30:47.470215 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31e3220_e28e_432d_9f9a_86ce56e66a78.slice/crio-3e04dfcf68e1a70892103116e8757b15a1856ae409a9dfed47e6d77a376c5ff7 WatchSource:0}: Error finding container 3e04dfcf68e1a70892103116e8757b15a1856ae409a9dfed47e6d77a376c5ff7: Status 404 returned error can't find the container with id 3e04dfcf68e1a70892103116e8757b15a1856ae409a9dfed47e6d77a376c5ff7 Oct 30 04:30:47.474874 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.474839 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp"] Oct 30 04:30:47.475311 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.475267 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:30:47.475404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.475356 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:30:47.475497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.475399 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:30:47.531935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.531903 4793 generic.go:356] "Generic (PLEG): container finished" podID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerID="a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40" exitCode=0 Oct 30 04:30:47.532042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.531973 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" event={"ID":"a64b9de2-a39e-4d00-8283-1fad481ca2d2","Type":"ContainerDied","Data":"a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40"} Oct 30 04:30:47.532529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.532508 4793 scope.go:122] "RemoveContainer" containerID="a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40" Oct 30 04:30:47.533446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.533396 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" event={"ID":"b31e3220-e28e-432d-9f9a-86ce56e66a78","Type":"ContainerStarted","Data":"3e04dfcf68e1a70892103116e8757b15a1856ae409a9dfed47e6d77a376c5ff7"} Oct 30 04:30:47.534408 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.534374 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:30:47.534525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.534505 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:30:47.534570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.534550 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:30:47.928568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:47.928520 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:48.245563 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:48.231203 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:48.542966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:48.542856 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" event={"ID":"b31e3220-e28e-432d-9f9a-86ce56e66a78","Type":"ContainerStarted","Data":"683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592"} Oct 30 04:30:48.544681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:48.544655 4793 generic.go:356] "Generic (PLEG): container finished" podID="62379909-e590-4685-9735-5f3def3f7d99" containerID="701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498" exitCode=0 Oct 30 04:30:48.544827 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:48.544724 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" event={"ID":"62379909-e590-4685-9735-5f3def3f7d99","Type":"ContainerDied","Data":"701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498"} Oct 30 04:30:48.545144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:48.545128 4793 scope.go:122] "RemoveContainer" containerID="701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498" Oct 30 04:30:48.546857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:48.546822 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:30:48.546939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:48.546907 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:30:48.546985 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:48.546954 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:30:48.547798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:48.547773 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" event={"ID":"a64b9de2-a39e-4d00-8283-1fad481ca2d2","Type":"ContainerStarted","Data":"4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4"} Oct 30 04:30:48.611185 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:48.611139 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" podStartSLOduration=2.61112388 podStartE2EDuration="2.61112388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:30:48.608934145 +0000 UTC m=+1971.894808920" watchObservedRunningTime="2026-10-30 04:30:48.61112388 +0000 UTC m=+1971.896998655" Oct 30 04:30:49.134093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.134059 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:49.139748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.139722 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:49.239732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.239697 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:49.354527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.354489 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:30:49.358556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.358522 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 04:30:49.358777 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.358749 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 04:30:49.448738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.448699 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:49.625544 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.625495 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" event={"ID":"62379909-e590-4685-9735-5f3def3f7d99","Type":"ContainerStarted","Data":"abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf"} Oct 30 04:30:49.626170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.626146 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:49.627099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.627080 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:30:49.637472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.637454 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:30:49.637664 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.637637 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 04:30:49.637747 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.637720 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 04:30:49.776499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:49.776399 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:50.372443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:50.353471 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 04:30:50.372443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:50.353542 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 04:30:50.638713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:50.638319 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 04:30:50.638713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:50.638373 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 04:30:51.363442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:51.357534 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 04:30:51.363442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:51.357607 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 04:30:51.447149 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:51.447114 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:51.638517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:51.638404 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 04:30:51.638517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:51.638486 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 04:30:51.702443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:51.695434 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:52.358546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:52.357769 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 04:30:52.358546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:52.357829 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 04:30:52.415449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:52.414802 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:52.635367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:52.635274 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:52.637627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:52.637590 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 04:30:52.637696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:52.637661 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 04:30:52.728655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:52.728621 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:30:53.356444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:53.353400 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 04:30:53.356444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:53.353507 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 04:30:53.603441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:53.579395 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:53.637878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:53.637794 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 04:30:53.637878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:53.637849 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 04:30:53.848639 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:53.844693 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:54.074182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:54.074057 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:54.439312 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:54.439280 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:30:54.441276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:54.441252 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:30:54.522259 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:54.522220 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:54.638639 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:54.638606 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 30 04:30:54.638826 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:54.638656 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 30 04:30:55.642771 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:55.642742 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:30:55.643180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:55.643111 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:30:55.643907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:55.643885 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:30:56.226378 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:56.226342 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:57.805755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:57.805711 4793 scope.go:122] "RemoveContainer" containerID="6a9e12199d1c64b9a16dd0ee9e7585b8826400659e69715e9a3b4ccf2ef5be54" Oct 30 04:30:57.869284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:57.869258 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:59.148468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:59.137139 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:30:59.378497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:30:59.378458 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:00.185459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:00.185428 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:02.079828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:02.079794 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:04.337217 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:04.337175 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:06.059136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:06.059104 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:07.807940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:07.807894 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:08.021938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:08.021905 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:09.081580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:09.081539 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:09.432855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:09.432812 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:09.654166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:09.654131 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:09.886112 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:09.886020 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:10.985333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:10.985299 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp"] Oct 30 04:31:10.991886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:10.991866 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.028908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.028875 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-eb9d23-serving-cert\"" Oct 30 04:31:11.049974 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.049925 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp"] Oct 30 04:31:11.064863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.064831 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-maas-api-tls\") pod \"maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.065057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.064907 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-metrics-certs\") pod \"maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.065057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.064944 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c1fe84-875c-4d1b-b083-6a368cbded98-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.065057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.065015 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmn6v\" (UniqueName: \"kubernetes.io/projected/e8c1fe84-875c-4d1b-b083-6a368cbded98-kube-api-access-tmn6v\") pod \"maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.166641 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.166612 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-maas-api-tls\") pod \"maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.166833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.166685 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-metrics-certs\") pod \"maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.166833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.166721 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c1fe84-875c-4d1b-b083-6a368cbded98-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.166833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.166787 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmn6v\" (UniqueName: \"kubernetes.io/projected/e8c1fe84-875c-4d1b-b083-6a368cbded98-kube-api-access-tmn6v\") pod \"maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.167559 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.167530 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c1fe84-875c-4d1b-b083-6a368cbded98-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.169200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.169176 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-metrics-certs\") pod \"maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.169345 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.169216 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-maas-api-tls\") pod \"maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.171335 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.171307 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:11.190319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.190291 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmn6v\" (UniqueName: \"kubernetes.io/projected/e8c1fe84-875c-4d1b-b083-6a368cbded98-kube-api-access-tmn6v\") pod \"maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.231973 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:31:11.231825 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:31:11.333636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.333540 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:11.389528 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.389499 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:11.667137 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:31:11.667102 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c1fe84_875c_4d1b_b083_6a368cbded98.slice/crio-6edb0357a86c712645813e36424d254fd76506ba695dd556ddce98f581534081 WatchSource:0}: Error finding container 6edb0357a86c712645813e36424d254fd76506ba695dd556ddce98f581534081: Status 404 returned error can't find the container with id 6edb0357a86c712645813e36424d254fd76506ba695dd556ddce98f581534081 Oct 30 04:31:11.669470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.669353 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp"] Oct 30 04:31:11.961167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:11.961131 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" event={"ID":"e8c1fe84-875c-4d1b-b083-6a368cbded98","Type":"ContainerStarted","Data":"6edb0357a86c712645813e36424d254fd76506ba695dd556ddce98f581534081"} Oct 30 04:31:12.970952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:12.970914 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" event={"ID":"e8c1fe84-875c-4d1b-b083-6a368cbded98","Type":"ContainerStarted","Data":"5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9"} Oct 30 04:31:12.971406 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:12.971000 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:13.018146 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:13.018103 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" podStartSLOduration=2.684328468 podStartE2EDuration="3.018089376s" totalImagesPullingTime="333.760908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:13.01418187 +0000 UTC m=+1996.300056645" watchObservedRunningTime="2026-10-30 04:31:13.018089376 +0000 UTC m=+1996.303964151" Oct 30 04:31:13.110908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:13.110879 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:13.476693 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:13.476655 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:13.700212 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:13.700163 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:14.925093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:14.925054 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:15.157719 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:15.157678 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:15.384138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:15.383469 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:16.722890 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:16.722855 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:18.406059 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.406000 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:18.750136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.750103 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62"] Oct 30 04:31:18.754287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.754264 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:18.776043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.776017 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62"] Oct 30 04:31:18.780241 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.780216 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-metrics-certs\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:18.780389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.780284 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:18.780389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.780363 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-maas-api-tls\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:18.780523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.780389 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99wn6\" (UniqueName: \"kubernetes.io/projected/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-kube-api-access-99wn6\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:18.781126 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.781102 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-05f827-serving-cert\"" Oct 30 04:31:18.881716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.881682 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:18.881955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.881925 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-maas-api-tls\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:18.882046 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.881977 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99wn6\" (UniqueName: \"kubernetes.io/projected/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-kube-api-access-99wn6\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:18.882108 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:31:18.882044 4793 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-serving-cert: secret "maas-api-e2e-praxis-b-05f827-serving-cert" not found Oct 30 04:31:18.882166 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:31:18.882142 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-maas-api-tls podName:fb8b5dc2-4bc4-4580-b82a-747d9ce48611 nodeName:}" failed. No retries permitted until 2026-10-30 04:31:19.382120817 +0000 UTC m=+2002.667995590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-maas-api-tls") pod "maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" (UID: "fb8b5dc2-4bc4-4580-b82a-747d9ce48611") : secret "maas-api-e2e-praxis-b-05f827-serving-cert" not found Oct 30 04:31:18.882338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.882318 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-metrics-certs\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:18.882651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.882628 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:18.884912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.884887 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-metrics-certs\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:18.907802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.907778 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99wn6\" (UniqueName: \"kubernetes.io/projected/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-kube-api-access-99wn6\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:18.946473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.946443 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:18.987047 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:18.987023 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:31:19.168993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:19.168907 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:19.391464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:19.391413 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-maas-api-tls\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:19.393900 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:19.393875 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-maas-api-tls\") pod \"maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:19.674664 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:19.674627 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:19.991121 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:31:19.991088 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8b5dc2_4bc4_4580_b82a_747d9ce48611.slice/crio-8880691aab83b82632653afa53cadc14ba93abda83a71cd91fe0e9e3d607b547 WatchSource:0}: Error finding container 8880691aab83b82632653afa53cadc14ba93abda83a71cd91fe0e9e3d607b547: Status 404 returned error can't find the container with id 8880691aab83b82632653afa53cadc14ba93abda83a71cd91fe0e9e3d607b547 Oct 30 04:31:19.991430 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:19.991395 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62"] Oct 30 04:31:20.039535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:20.039500 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" event={"ID":"fb8b5dc2-4bc4-4580-b82a-747d9ce48611","Type":"ContainerStarted","Data":"8880691aab83b82632653afa53cadc14ba93abda83a71cd91fe0e9e3d607b547"} Oct 30 04:31:20.641948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:20.641911 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:21.051955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:21.051882 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" event={"ID":"fb8b5dc2-4bc4-4580-b82a-747d9ce48611","Type":"ContainerStarted","Data":"63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3"} Oct 30 04:31:21.075813 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:21.075786 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:21.141830 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:21.141759 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" podStartSLOduration=2.731243985 podStartE2EDuration="3.141738027s" totalImagesPullingTime="410.494042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:21.117786545 +0000 UTC m=+2004.403661321" watchObservedRunningTime="2026-10-30 04:31:21.141738027 +0000 UTC m=+2004.427612802" Oct 30 04:31:22.428800 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:22.428751 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:23.934517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:23.934481 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg"] Oct 30 04:31:24.150367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:24.150316 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:24.348371 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:24.348271 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:25.096023 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:25.095988 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7"] Oct 30 04:31:25.096467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:25.096242 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" podUID="62a6fc57-013f-4b08-8768-f01a594f6c64" containerName="istio-proxy" containerID="cri-o://31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7" gracePeriod=30 Oct 30 04:31:26.073234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:26.073200 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:28.092333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:28.092306 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:31:28.227220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:28.227185 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:28.935343 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:28.935308 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" podUID="7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9" containerName="payload-pre-processing" containerID="cri-o://24de24043639f1e9b03045d8ba21fa0fbf7d5cc0a57e728275f90a14e0e2e462" gracePeriod=25 Oct 30 04:31:29.144438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:29.144392 4793 generic.go:356] "Generic (PLEG): container finished" podID="7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9" containerID="24de24043639f1e9b03045d8ba21fa0fbf7d5cc0a57e728275f90a14e0e2e462" exitCode=0 Oct 30 04:31:29.144869 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:29.144447 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" event={"ID":"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9","Type":"ContainerDied","Data":"24de24043639f1e9b03045d8ba21fa0fbf7d5cc0a57e728275f90a14e0e2e462"} Oct 30 04:31:29.267878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:29.267768 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" Oct 30 04:31:29.401881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:29.401849 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-plugins-config-volume\") pod \"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9\" (UID: \"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9\") " Oct 30 04:31:29.402091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:29.401985 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-kube-api-access-6xhh7\" (UniqueName: \"kubernetes.io/projected/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-kube-api-access-6xhh7\") pod \"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9\" (UID: \"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9\") " Oct 30 04:31:29.402316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:29.402285 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-plugins-config-volume" pod "7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9" (UID: "7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:31:29.402686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:29.402668 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:29.404139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:29.404115 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-kube-api-access-6xhh7" pod "7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9" (UID: "7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9"). InnerVolumeSpecName "kube-api-access-6xhh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:31:29.504356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:29.504261 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xhh7\" (UniqueName: \"kubernetes.io/projected/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9-kube-api-access-6xhh7\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:29.889375 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:29.888708 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:30.155077 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.155016 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" event={"ID":"7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9","Type":"ContainerDied","Data":"73b8d604f97f6a67be15b51be9fd5dbe88e01658be5845fd040de0be3c2f4737"} Oct 30 04:31:30.155077 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.155050 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg" Oct 30 04:31:30.155077 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.155077 4793 scope.go:122] "RemoveContainer" containerID="24de24043639f1e9b03045d8ba21fa0fbf7d5cc0a57e728275f90a14e0e2e462" Oct 30 04:31:30.224921 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.224892 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg"] Oct 30 04:31:30.236951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.236925 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-7c546b744c-snwbg"] Oct 30 04:31:30.378978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.378956 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:31:30.519875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.519842 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-envoy\") pod \"62a6fc57-013f-4b08-8768-f01a594f6c64\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " Oct 30 04:31:30.520054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.519971 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-podinfo\") pod \"62a6fc57-013f-4b08-8768-f01a594f6c64\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " Oct 30 04:31:30.520054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520011 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-socket\") pod \"62a6fc57-013f-4b08-8768-f01a594f6c64\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " Oct 30 04:31:30.520054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520040 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a6fc57-013f-4b08-8768-f01a594f6c64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a6fc57-013f-4b08-8768-f01a594f6c64-istiod-ca-cert\") pod \"62a6fc57-013f-4b08-8768-f01a594f6c64\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " Oct 30 04:31:30.520227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520074 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-certs\") pod \"62a6fc57-013f-4b08-8768-f01a594f6c64\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " Oct 30 04:31:30.520227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520098 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-kube-api-access-4wnch\" (UniqueName: \"kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-kube-api-access-4wnch\") pod \"62a6fc57-013f-4b08-8768-f01a594f6c64\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " Oct 30 04:31:30.520227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520123 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-data\") pod \"62a6fc57-013f-4b08-8768-f01a594f6c64\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " Oct 30 04:31:30.520227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520156 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-token\" (UniqueName: \"kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-token\") pod \"62a6fc57-013f-4b08-8768-f01a594f6c64\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " Oct 30 04:31:30.520227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520190 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-credential-socket\") pod \"62a6fc57-013f-4b08-8768-f01a594f6c64\" (UID: \"62a6fc57-013f-4b08-8768-f01a594f6c64\") " Oct 30 04:31:30.520518 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520284 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-socket" pod "62a6fc57-013f-4b08-8768-f01a594f6c64" (UID: "62a6fc57-013f-4b08-8768-f01a594f6c64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:31:30.520652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520629 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-credential-socket" pod "62a6fc57-013f-4b08-8768-f01a594f6c64" (UID: "62a6fc57-013f-4b08-8768-f01a594f6c64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:31:30.520796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520780 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:30.520863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520801 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:30.520918 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520868 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a6fc57-013f-4b08-8768-f01a594f6c64-istiod-ca-cert" pod "62a6fc57-013f-4b08-8768-f01a594f6c64" (UID: "62a6fc57-013f-4b08-8768-f01a594f6c64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:31:30.521109 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.521084 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-data" pod "62a6fc57-013f-4b08-8768-f01a594f6c64" (UID: "62a6fc57-013f-4b08-8768-f01a594f6c64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:31:30.521191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.520566 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-certs" pod "62a6fc57-013f-4b08-8768-f01a594f6c64" (UID: "62a6fc57-013f-4b08-8768-f01a594f6c64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:31:30.522546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.522521 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-kube-api-access-4wnch" pod "62a6fc57-013f-4b08-8768-f01a594f6c64" (UID: "62a6fc57-013f-4b08-8768-f01a594f6c64"). InnerVolumeSpecName "kube-api-access-4wnch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:31:30.522710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.522690 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-podinfo" pod "62a6fc57-013f-4b08-8768-f01a594f6c64" (UID: "62a6fc57-013f-4b08-8768-f01a594f6c64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:31:30.522761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.522728 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-envoy" pod "62a6fc57-013f-4b08-8768-f01a594f6c64" (UID: "62a6fc57-013f-4b08-8768-f01a594f6c64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:31:30.522882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.522862 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-token" pod "62a6fc57-013f-4b08-8768-f01a594f6c64" (UID: "62a6fc57-013f-4b08-8768-f01a594f6c64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:31:30.544580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.544549 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:31:30.544734 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.544606 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:31:30.622301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.622270 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62a6fc57-013f-4b08-8768-f01a594f6c64-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:30.622301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.622302 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:30.622513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.622315 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wnch\" (UniqueName: \"kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-kube-api-access-4wnch\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:30.622513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.622327 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:30.622513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.622339 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:30.622513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.622350 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:30.622513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.622361 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62a6fc57-013f-4b08-8768-f01a594f6c64-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:30.861518 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:30.861273 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9" path="/var/lib/kubelet/pods/7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9/volumes" Oct 30 04:31:31.166940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:31.166858 4793 generic.go:356] "Generic (PLEG): container finished" podID="62a6fc57-013f-4b08-8768-f01a594f6c64" containerID="31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7" exitCode=0 Oct 30 04:31:31.166940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:31.166933 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" Oct 30 04:31:31.167516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:31.166939 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" event={"ID":"62a6fc57-013f-4b08-8768-f01a594f6c64","Type":"ContainerDied","Data":"31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7"} Oct 30 04:31:31.167516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:31.167003 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7" event={"ID":"62a6fc57-013f-4b08-8768-f01a594f6c64","Type":"ContainerDied","Data":"6461a3aa843dd4234c3abab62fccb653c20cd25ba154696e33597db26b4f4e52"} Oct 30 04:31:31.167516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:31.167032 4793 scope.go:122] "RemoveContainer" containerID="31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7" Oct 30 04:31:31.181265 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:31.181246 4793 scope.go:122] "RemoveContainer" containerID="31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7" Oct 30 04:31:31.181585 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:31:31.181563 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7\": container with ID starting with 31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7 not found: ID does not exist" containerID="31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7" Oct 30 04:31:31.181647 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:31.181596 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7"} err="failed to get container status \"31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7\": rpc error: code = NotFound desc = could not find container \"31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7\": container with ID starting with 31f6c9991197c1f4b4dc4d86a5ae01e4c2a5ae41b02cc9aa5faf8011fe710ed7 not found: ID does not exist" Oct 30 04:31:31.237062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:31.237034 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7"] Oct 30 04:31:31.256512 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:31.256484 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-hpmf7"] Oct 30 04:31:32.050217 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:32.050134 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:32.182785 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:32.182756 4793 generic.go:356] "Generic (PLEG): container finished" podID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerID="7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50" exitCode=0 Oct 30 04:31:32.183324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:32.182831 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" event={"ID":"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a","Type":"ContainerDied","Data":"7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50"} Oct 30 04:31:32.183393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:32.183302 4793 scope.go:122] "RemoveContainer" containerID="7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50" Oct 30 04:31:32.185020 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:32.184988 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:31:32.185132 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:32.185058 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:31:32.185132 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:32.185091 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:31:32.846862 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:32.846830 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a6fc57-013f-4b08-8768-f01a594f6c64" path="/var/lib/kubelet/pods/62a6fc57-013f-4b08-8768-f01a594f6c64/volumes" Oct 30 04:31:33.196891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:33.196854 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" event={"ID":"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a","Type":"ContainerStarted","Data":"ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3"} Oct 30 04:31:33.420986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:33.420951 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:31:33.421188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:33.421162 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 04:31:33.421251 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:33.421220 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 04:31:33.697690 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:33.697646 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:34.010610 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:34.010526 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:34.221946 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:34.221910 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:34.420113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:34.420037 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 04:31:34.420113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:34.420087 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 04:31:35.402984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:35.402951 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:35.421101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:35.421072 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 04:31:35.421282 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:35.421118 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 04:31:36.420555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:36.420518 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 04:31:36.421029 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:36.420567 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 04:31:37.007902 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:37.007864 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:37.421151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:37.421060 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 30 04:31:37.421151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:37.421130 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 30 04:31:38.424201 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:38.424172 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:31:38.424649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:38.424630 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:31:38.425258 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:38.425239 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:31:38.687220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:38.687173 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:40.323378 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:40.323300 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:40.525700 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:40.525667 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:40.735815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:40.735781 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:41.219628 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:31:41.219591 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:31:42.189279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:42.189237 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:42.462498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:42.462028 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:44.227888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:44.227843 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:44.681915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:44.681713 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s"] Oct 30 04:31:44.682143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:44.681978 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" podUID="41615d2b-3148-4a60-a6a0-4ad6191bbd4e" containerName="istio-proxy" containerID="cri-o://978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa" gracePeriod=30 Oct 30 04:31:45.926813 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:45.926772 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:47.482980 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:47.482939 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:48.061148 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:48.061112 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:49.861297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:49.861205 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:50.013629 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.013603 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:31:50.074123 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.074081 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-credential-socket\") pod \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " Oct 30 04:31:50.074310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.074153 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istiod-ca-cert\") pod \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " Oct 30 04:31:50.074310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.074202 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-podinfo\") pod \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " Oct 30 04:31:50.074476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.074313 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-certs\") pod \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " Oct 30 04:31:50.074476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.074365 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-socket\") pod \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " Oct 30 04:31:50.074476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.074402 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-envoy\") pod \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " Oct 30 04:31:50.074638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.074521 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-token\" (UniqueName: \"kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-token\") pod \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " Oct 30 04:31:50.074638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.074564 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-data\") pod \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " Oct 30 04:31:50.074638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.074616 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-kube-api-access-66jr8\" (UniqueName: \"kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-kube-api-access-66jr8\") pod \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\" (UID: \"41615d2b-3148-4a60-a6a0-4ad6191bbd4e\") " Oct 30 04:31:50.075856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.075827 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-certs" pod "41615d2b-3148-4a60-a6a0-4ad6191bbd4e" (UID: "41615d2b-3148-4a60-a6a0-4ad6191bbd4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:31:50.076481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.076277 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-credential-socket" pod "41615d2b-3148-4a60-a6a0-4ad6191bbd4e" (UID: "41615d2b-3148-4a60-a6a0-4ad6191bbd4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:31:50.076594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.076427 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istiod-ca-cert" pod "41615d2b-3148-4a60-a6a0-4ad6191bbd4e" (UID: "41615d2b-3148-4a60-a6a0-4ad6191bbd4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:31:50.076992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.076736 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-socket" pod "41615d2b-3148-4a60-a6a0-4ad6191bbd4e" (UID: "41615d2b-3148-4a60-a6a0-4ad6191bbd4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:31:50.079564 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.079529 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-data" pod "41615d2b-3148-4a60-a6a0-4ad6191bbd4e" (UID: "41615d2b-3148-4a60-a6a0-4ad6191bbd4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:31:50.080804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.080782 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-envoy" pod "41615d2b-3148-4a60-a6a0-4ad6191bbd4e" (UID: "41615d2b-3148-4a60-a6a0-4ad6191bbd4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:31:50.092948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.092897 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-token" pod "41615d2b-3148-4a60-a6a0-4ad6191bbd4e" (UID: "41615d2b-3148-4a60-a6a0-4ad6191bbd4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:31:50.094817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.094785 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-kube-api-access-66jr8" pod "41615d2b-3148-4a60-a6a0-4ad6191bbd4e" (UID: "41615d2b-3148-4a60-a6a0-4ad6191bbd4e"). InnerVolumeSpecName "kube-api-access-66jr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:31:50.095087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.095056 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-podinfo" pod "41615d2b-3148-4a60-a6a0-4ad6191bbd4e" (UID: "41615d2b-3148-4a60-a6a0-4ad6191bbd4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:31:50.176604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.176565 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:50.176604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.176597 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:50.176604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.176610 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:50.176855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.176623 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:50.176855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.176633 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:50.176855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.176644 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66jr8\" (UniqueName: \"kubernetes.io/projected/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-kube-api-access-66jr8\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:50.176855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.176657 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:50.176855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.176669 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:50.176855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.176680 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41615d2b-3148-4a60-a6a0-4ad6191bbd4e-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:50.368267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.368176 4793 generic.go:356] "Generic (PLEG): container finished" podID="41615d2b-3148-4a60-a6a0-4ad6191bbd4e" containerID="978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa" exitCode=0 Oct 30 04:31:50.368480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.368290 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" event={"ID":"41615d2b-3148-4a60-a6a0-4ad6191bbd4e","Type":"ContainerDied","Data":"978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa"} Oct 30 04:31:50.368480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.368372 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" event={"ID":"41615d2b-3148-4a60-a6a0-4ad6191bbd4e","Type":"ContainerDied","Data":"b2e2fed709f1b606937bbb6890a29cbe9bdbf28982b31db9a4cb959320a50f17"} Oct 30 04:31:50.368480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.368404 4793 scope.go:122] "RemoveContainer" containerID="978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa" Oct 30 04:31:50.368677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.368575 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" Oct 30 04:31:50.384084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.384060 4793 scope.go:122] "RemoveContainer" containerID="978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa" Oct 30 04:31:50.384467 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:31:50.384403 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa\": container with ID starting with 978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa not found: ID does not exist" containerID="978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa" Oct 30 04:31:50.384607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.384478 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa"} err="failed to get container status \"978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa\": rpc error: code = NotFound desc = could not find container \"978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa\": container with ID starting with 978291fb7b5330accd10cc1d58c04f4ca83275ed57e87ee1bdb62e8a7e0db5fa not found: ID does not exist" Oct 30 04:31:50.426108 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.426073 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s"] Oct 30 04:31:50.429466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.429442 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s"] Oct 30 04:31:50.666641 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.666549 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:31:50.666807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.666662 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:31:50.851292 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.851259 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41615d2b-3148-4a60-a6a0-4ad6191bbd4e" path="/var/lib/kubelet/pods/41615d2b-3148-4a60-a6a0-4ad6191bbd4e/volumes" Oct 30 04:31:50.900568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.900535 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:31:50.901030 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:50.900588 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-z2g7s" podUID="41615d2b-3148-4a60-a6a0-4ad6191bbd4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:31:51.220317 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.220268 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:51.424447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.424379 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:51.528559 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.528473 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt"] Oct 30 04:31:51.533441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.529876 4793 cpu_manager.go:414] "containerMap: removing container" podUID="7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9" containerName="payload-pre-processing" Oct 30 04:31:51.533650 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.533629 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9" containerName="payload-pre-processing" Oct 30 04:31:51.533751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.533739 4793 cpu_manager.go:414] "containerMap: removing container" podUID="62a6fc57-013f-4b08-8768-f01a594f6c64" containerName="istio-proxy" Oct 30 04:31:51.545228 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.545199 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a6fc57-013f-4b08-8768-f01a594f6c64" containerName="istio-proxy" Oct 30 04:31:51.545462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.545444 4793 cpu_manager.go:414] "containerMap: removing container" podUID="41615d2b-3148-4a60-a6a0-4ad6191bbd4e" containerName="istio-proxy" Oct 30 04:31:51.545595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.545581 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41615d2b-3148-4a60-a6a0-4ad6191bbd4e" containerName="istio-proxy" Oct 30 04:31:51.545978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.545958 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a0b36cb-e690-40c4-b7ea-57e37b2e5bc9" containerName="payload-pre-processing" Oct 30 04:31:51.546115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.546102 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a6fc57-013f-4b08-8768-f01a594f6c64" containerName="istio-proxy" Oct 30 04:31:51.550698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.546201 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="41615d2b-3148-4a60-a6a0-4ad6191bbd4e" containerName="istio-proxy" Oct 30 04:31:51.581996 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.581965 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt"] Oct 30 04:31:51.582294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.582281 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" Oct 30 04:31:51.610998 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.610966 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6a777c-dockercfg-zx2zr\"" Oct 30 04:31:51.611143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.611034 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6a777c\"" Oct 30 04:31:51.619638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.619613 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt\" (UID: \"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" Oct 30 04:31:51.619866 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.619849 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnxsz\" (UniqueName: \"kubernetes.io/projected/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-kube-api-access-vnxsz\") pod \"payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt\" (UID: \"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" Oct 30 04:31:51.721294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.721259 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnxsz\" (UniqueName: \"kubernetes.io/projected/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-kube-api-access-vnxsz\") pod \"payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt\" (UID: \"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" Oct 30 04:31:51.721480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.721326 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt\" (UID: \"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" Oct 30 04:31:51.722066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.722044 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt\" (UID: \"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" Oct 30 04:31:51.743740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.743711 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg"] Oct 30 04:31:51.750130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.750104 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" Oct 30 04:31:51.753356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.753328 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnxsz\" (UniqueName: \"kubernetes.io/projected/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-kube-api-access-vnxsz\") pod \"payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt\" (UID: \"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" Oct 30 04:31:51.772458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.772430 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg"] Oct 30 04:31:51.822572 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.822530 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgxdc\" (UniqueName: \"kubernetes.io/projected/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-kube-api-access-lgxdc\") pod \"payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg\" (UID: \"859cce50-f6af-42e6-a3d6-9a4c76f04c7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" Oct 30 04:31:51.822732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.822711 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg\" (UID: \"859cce50-f6af-42e6-a3d6-9a4c76f04c7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" Oct 30 04:31:51.907801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.907775 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" Oct 30 04:31:51.924167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.924136 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgxdc\" (UniqueName: \"kubernetes.io/projected/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-kube-api-access-lgxdc\") pod \"payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg\" (UID: \"859cce50-f6af-42e6-a3d6-9a4c76f04c7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" Oct 30 04:31:51.924409 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.924385 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg\" (UID: \"859cce50-f6af-42e6-a3d6-9a4c76f04c7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" Oct 30 04:31:51.925002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.924974 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg\" (UID: \"859cce50-f6af-42e6-a3d6-9a4c76f04c7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" Oct 30 04:31:51.948373 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:51.948345 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgxdc\" (UniqueName: \"kubernetes.io/projected/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-kube-api-access-lgxdc\") pod \"payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg\" (UID: \"859cce50-f6af-42e6-a3d6-9a4c76f04c7f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" Oct 30 04:31:52.109912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.109883 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" Oct 30 04:31:52.231865 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:31:52.231823 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5cdbeb_cea3_4d84_befb_57c5fe7c6b73.slice/crio-cbbc92151524b41b2ef554e74dc9e64e1808dd3c58cad0d97b36901d158abd66 WatchSource:0}: Error finding container cbbc92151524b41b2ef554e74dc9e64e1808dd3c58cad0d97b36901d158abd66: Status 404 returned error can't find the container with id cbbc92151524b41b2ef554e74dc9e64e1808dd3c58cad0d97b36901d158abd66 Oct 30 04:31:52.234828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.234804 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt"] Oct 30 04:31:52.391023 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.390942 4793 generic.go:356] "Generic (PLEG): container finished" podID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerID="683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592" exitCode=0 Oct 30 04:31:52.391187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.391017 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" event={"ID":"b31e3220-e28e-432d-9f9a-86ce56e66a78","Type":"ContainerDied","Data":"683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592"} Oct 30 04:31:52.391570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.391550 4793 scope.go:122] "RemoveContainer" containerID="683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592" Oct 30 04:31:52.393456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.392878 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" event={"ID":"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73","Type":"ContainerStarted","Data":"8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942"} Oct 30 04:31:52.393456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.392927 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" event={"ID":"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73","Type":"ContainerStarted","Data":"cbbc92151524b41b2ef554e74dc9e64e1808dd3c58cad0d97b36901d158abd66"} Oct 30 04:31:52.393456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.393011 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" Oct 30 04:31:52.393613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.393562 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:31:52.393663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.393643 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:31:52.393714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.393699 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:31:52.411911 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:31:52.411885 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859cce50_f6af_42e6_a3d6_9a4c76f04c7f.slice/crio-4a9295dd621160364a2c40f64f6d2b195422dd510a95cc54ac52362f2abbe5bf WatchSource:0}: Error finding container 4a9295dd621160364a2c40f64f6d2b195422dd510a95cc54ac52362f2abbe5bf: Status 404 returned error can't find the container with id 4a9295dd621160364a2c40f64f6d2b195422dd510a95cc54ac52362f2abbe5bf Oct 30 04:31:52.427059 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.425861 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg"] Oct 30 04:31:52.446762 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.446720 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:52.496457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.493156 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" podStartSLOduration=1.493136357 podStartE2EDuration="1.493136357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:52.491839392 +0000 UTC m=+2035.777714190" watchObservedRunningTime="2026-10-30 04:31:52.493136357 +0000 UTC m=+2035.779011134" Oct 30 04:31:52.659947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:52.659859 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:53.404790 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:53.404183 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" event={"ID":"859cce50-f6af-42e6-a3d6-9a4c76f04c7f","Type":"ContainerStarted","Data":"06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665"} Oct 30 04:31:53.404790 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:53.404248 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" event={"ID":"859cce50-f6af-42e6-a3d6-9a4c76f04c7f","Type":"ContainerStarted","Data":"4a9295dd621160364a2c40f64f6d2b195422dd510a95cc54ac52362f2abbe5bf"} Oct 30 04:31:53.404790 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:53.404525 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" Oct 30 04:31:53.406990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:53.406965 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" event={"ID":"b31e3220-e28e-432d-9f9a-86ce56e66a78","Type":"ContainerStarted","Data":"156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730"} Oct 30 04:31:53.448371 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:53.448324 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" podStartSLOduration=2.448311881 podStartE2EDuration="2.448311881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:31:53.44604389 +0000 UTC m=+2036.731918667" watchObservedRunningTime="2026-10-30 04:31:53.448311881 +0000 UTC m=+2036.734186686" Oct 30 04:31:53.952562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:53.952528 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:54.134073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:54.133972 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:31:54.134284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:54.134233 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 04:31:54.134372 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:54.134288 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 04:31:54.196455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:54.192380 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:54.405652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:54.405552 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z"] Oct 30 04:31:54.649438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:54.647303 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:54.853338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:54.853304 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:55.092443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:55.087167 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:55.144523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:55.142899 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 04:31:55.144523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:55.142957 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 04:31:56.016339 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:56.016302 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:56.133544 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:56.133514 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 04:31:56.133718 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:56.133561 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 04:31:57.133891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:57.133860 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 04:31:57.134322 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:57.133920 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 04:31:57.876448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:57.871458 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw"] Oct 30 04:31:57.876448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:57.871745 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" podUID="ccc2c258-ffe4-4e16-bf5e-996d41b90f45" containerName="maas-api" containerID="cri-o://5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d" gracePeriod=30 Oct 30 04:31:57.905442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:57.896197 4793 scope.go:122] "RemoveContainer" containerID="0f08333d21f8792cc1589241a4a7ff207e1e2a25574c4e49850606f4c3dea852" Oct 30 04:31:57.997338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:57.996949 4793 scope.go:122] "RemoveContainer" containerID="06db8207b43b40f455bbb94836bc98806be8371053e86b10928f63502124835e" Oct 30 04:31:58.009227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.009205 4793 scope.go:122] "RemoveContainer" containerID="7ff5ad862c0ae0d369c1e9d138b0eeffb44cd579a5aebac7728ac99911f8064f" Oct 30 04:31:58.024054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.024031 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:58.143947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.138014 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 04:31:58.143947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.138089 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 04:31:58.365934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.365603 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:31:58.449824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.449786 4793 generic.go:356] "Generic (PLEG): container finished" podID="ccc2c258-ffe4-4e16-bf5e-996d41b90f45" containerID="5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d" exitCode=0 Oct 30 04:31:58.450026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.449854 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" Oct 30 04:31:58.450026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.449877 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" event={"ID":"ccc2c258-ffe4-4e16-bf5e-996d41b90f45","Type":"ContainerDied","Data":"5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d"} Oct 30 04:31:58.450026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.449938 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw" event={"ID":"ccc2c258-ffe4-4e16-bf5e-996d41b90f45","Type":"ContainerDied","Data":"c210664a9481b504111508a0d189fff3f8dbcc047d43b6e90508c30c578f26a4"} Oct 30 04:31:58.450026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.449965 4793 scope.go:122] "RemoveContainer" containerID="5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d" Oct 30 04:31:58.472513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.469885 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-maas-api-tls\") pod \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " Oct 30 04:31:58.472513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.470031 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-openshift-service-ca\") pod \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " Oct 30 04:31:58.472513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.470134 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-metrics-certs\") pod \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " Oct 30 04:31:58.472513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.470176 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-kube-api-access-qftvk\" (UniqueName: \"kubernetes.io/projected/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-kube-api-access-qftvk\") pod \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\" (UID: \"ccc2c258-ffe4-4e16-bf5e-996d41b90f45\") " Oct 30 04:31:58.486463 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.485864 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-metrics-certs" pod "ccc2c258-ffe4-4e16-bf5e-996d41b90f45" (UID: "ccc2c258-ffe4-4e16-bf5e-996d41b90f45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:31:58.486704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.486654 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-openshift-service-ca" pod "ccc2c258-ffe4-4e16-bf5e-996d41b90f45" (UID: "ccc2c258-ffe4-4e16-bf5e-996d41b90f45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:31:58.487852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.487808 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-kube-api-access-qftvk" pod "ccc2c258-ffe4-4e16-bf5e-996d41b90f45" (UID: "ccc2c258-ffe4-4e16-bf5e-996d41b90f45"). InnerVolumeSpecName "kube-api-access-qftvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:31:58.487989 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.487965 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-maas-api-tls" pod "ccc2c258-ffe4-4e16-bf5e-996d41b90f45" (UID: "ccc2c258-ffe4-4e16-bf5e-996d41b90f45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:31:58.489305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.489286 4793 scope.go:122] "RemoveContainer" containerID="5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d" Oct 30 04:31:58.489703 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:31:58.489677 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d\": container with ID starting with 5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d not found: ID does not exist" containerID="5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d" Oct 30 04:31:58.489799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.489716 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d"} err="failed to get container status \"5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d\": rpc error: code = NotFound desc = could not find container \"5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d\": container with ID starting with 5bc278d73886ad7db405208e0a02c2121d527b3231b0ae4af6c79b8554a7897d not found: ID does not exist" Oct 30 04:31:58.571716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.571684 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:58.571716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.571714 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:58.571716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.571725 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:58.571958 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.571736 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qftvk\" (UniqueName: \"kubernetes.io/projected/ccc2c258-ffe4-4e16-bf5e-996d41b90f45-kube-api-access-qftvk\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:58.727965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.727862 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:31:58.801148 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.801116 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw"] Oct 30 04:31:58.804440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.804403 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-58cdc8-b48dbcdc7-g7kgw"] Oct 30 04:31:58.847549 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:58.847507 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc2c258-ffe4-4e16-bf5e-996d41b90f45" path="/var/lib/kubelet/pods/ccc2c258-ffe4-4e16-bf5e-996d41b90f45/volumes" Oct 30 04:31:59.137582 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:59.137513 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:31:59.137815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:59.137795 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:31:59.138596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:59.138577 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:31:59.406718 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:59.406627 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" podUID="23e14557-01b4-4800-980a-0ab377967fcf" containerName="payload-processing" containerID="cri-o://b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8" gracePeriod=25 Oct 30 04:31:59.687122 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:59.687100 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" Oct 30 04:31:59.786483 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:59.786450 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23e14557-01b4-4800-980a-0ab377967fcf-kube-api-access-xz2r4\" (UniqueName: \"kubernetes.io/projected/23e14557-01b4-4800-980a-0ab377967fcf-kube-api-access-xz2r4\") pod \"23e14557-01b4-4800-980a-0ab377967fcf\" (UID: \"23e14557-01b4-4800-980a-0ab377967fcf\") " Oct 30 04:31:59.786661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:59.786528 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23e14557-01b4-4800-980a-0ab377967fcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23e14557-01b4-4800-980a-0ab377967fcf-plugins-config-volume\") pod \"23e14557-01b4-4800-980a-0ab377967fcf\" (UID: \"23e14557-01b4-4800-980a-0ab377967fcf\") " Oct 30 04:31:59.786876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:59.786850 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23e14557-01b4-4800-980a-0ab377967fcf-plugins-config-volume" pod "23e14557-01b4-4800-980a-0ab377967fcf" (UID: "23e14557-01b4-4800-980a-0ab377967fcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:31:59.787000 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:59.786985 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23e14557-01b4-4800-980a-0ab377967fcf-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:31:59.788998 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:59.788969 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e14557-01b4-4800-980a-0ab377967fcf-kube-api-access-xz2r4" pod "23e14557-01b4-4800-980a-0ab377967fcf" (UID: "23e14557-01b4-4800-980a-0ab377967fcf"). InnerVolumeSpecName "kube-api-access-xz2r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:31:59.888393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:31:59.888363 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz2r4\" (UniqueName: \"kubernetes.io/projected/23e14557-01b4-4800-980a-0ab377967fcf-kube-api-access-xz2r4\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:00.469587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:00.469549 4793 generic.go:356] "Generic (PLEG): container finished" podID="23e14557-01b4-4800-980a-0ab377967fcf" containerID="b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8" exitCode=0 Oct 30 04:32:00.469587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:00.469590 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" Oct 30 04:32:00.470137 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:00.469629 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" event={"ID":"23e14557-01b4-4800-980a-0ab377967fcf","Type":"ContainerDied","Data":"b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8"} Oct 30 04:32:00.470137 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:00.469691 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z" event={"ID":"23e14557-01b4-4800-980a-0ab377967fcf","Type":"ContainerDied","Data":"52fe65d01111d0600216297ed843f2198a801d21e954fd965b5c4963bd3c5b65"} Oct 30 04:32:00.470137 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:00.469721 4793 scope.go:122] "RemoveContainer" containerID="b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8" Oct 30 04:32:00.487803 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:00.487776 4793 scope.go:122] "RemoveContainer" containerID="b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8" Oct 30 04:32:00.488663 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:00.488636 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8\": container with ID starting with b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8 not found: ID does not exist" containerID="b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8" Oct 30 04:32:00.488800 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:00.488670 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8"} err="failed to get container status \"b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8\": rpc error: code = NotFound desc = could not find container \"b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8\": container with ID starting with b0fc042297dc0b238298ae7edc55257a83e327644b90d301d141719d81502cc8 not found: ID does not exist" Oct 30 04:32:00.529869 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:00.529839 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z"] Oct 30 04:32:00.539813 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:00.539791 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-58cdc8-697cf6d99d-hhg4z"] Oct 30 04:32:00.832475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:00.832377 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:00.851637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:00.851597 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23e14557-01b4-4800-980a-0ab377967fcf" path="/var/lib/kubelet/pods/23e14557-01b4-4800-980a-0ab377967fcf/volumes" Oct 30 04:32:01.070175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:01.070125 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:01.286538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:01.286503 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:01.623860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:01.623770 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:01.819105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:01.819068 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:02.092441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:02.092160 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m"] Oct 30 04:32:02.092650 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:02.092458 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerName="istio-proxy" containerID="cri-o://4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4" gracePeriod=30 Oct 30 04:32:03.409119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:03.409093 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" Oct 30 04:32:03.432631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:03.432599 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:04.049706 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.049671 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh"] Oct 30 04:32:04.049975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.049948 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" podUID="ea8a64a8-e2ff-4427-85b7-8d559346a8a8" containerName="maas-api" containerID="cri-o://16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205" gracePeriod=30 Oct 30 04:32:04.194075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.194040 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:04.381314 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.381289 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:32:04.417324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.417285 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" Oct 30 04:32:04.484931 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.484893 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-kube-api-access-56htq\" (UniqueName: \"kubernetes.io/projected/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-kube-api-access-56htq\") pod \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " Oct 30 04:32:04.493838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.493804 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-maas-api-tls\") pod \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " Oct 30 04:32:04.494227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.494178 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-metrics-certs\") pod \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " Oct 30 04:32:04.494528 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.494495 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-openshift-service-ca\") pod \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\" (UID: \"ea8a64a8-e2ff-4427-85b7-8d559346a8a8\") " Oct 30 04:32:04.507200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.507170 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-openshift-service-ca" pod "ea8a64a8-e2ff-4427-85b7-8d559346a8a8" (UID: "ea8a64a8-e2ff-4427-85b7-8d559346a8a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:04.561755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.556794 4793 generic.go:356] "Generic (PLEG): container finished" podID="ea8a64a8-e2ff-4427-85b7-8d559346a8a8" containerID="16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205" exitCode=0 Oct 30 04:32:04.561755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.556965 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" event={"ID":"ea8a64a8-e2ff-4427-85b7-8d559346a8a8","Type":"ContainerDied","Data":"16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205"} Oct 30 04:32:04.561755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.557017 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" event={"ID":"ea8a64a8-e2ff-4427-85b7-8d559346a8a8","Type":"ContainerDied","Data":"e945bea7a5dedbeaed592a1112d04dbab32a09d56915616bd495069bb697fe20"} Oct 30 04:32:04.561755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.557046 4793 scope.go:122] "RemoveContainer" containerID="16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205" Oct 30 04:32:04.561755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.557186 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh" Oct 30 04:32:04.563778 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.563578 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-kube-api-access-56htq" pod "ea8a64a8-e2ff-4427-85b7-8d559346a8a8" (UID: "ea8a64a8-e2ff-4427-85b7-8d559346a8a8"). InnerVolumeSpecName "kube-api-access-56htq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:04.563778 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.563698 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-metrics-certs" pod "ea8a64a8-e2ff-4427-85b7-8d559346a8a8" (UID: "ea8a64a8-e2ff-4427-85b7-8d559346a8a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:04.569974 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.569619 4793 scope.go:122] "RemoveContainer" containerID="16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205" Oct 30 04:32:04.569974 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:04.569940 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205\": container with ID starting with 16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205 not found: ID does not exist" containerID="16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205" Oct 30 04:32:04.570119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.569979 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205"} err="failed to get container status \"16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205\": rpc error: code = NotFound desc = could not find container \"16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205\": container with ID starting with 16a5c26c26252b3646c7d2901defcc08089a118652f94f6b1a58806712819205 not found: ID does not exist" Oct 30 04:32:04.570200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.570175 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-maas-api-tls" pod "ea8a64a8-e2ff-4427-85b7-8d559346a8a8" (UID: "ea8a64a8-e2ff-4427-85b7-8d559346a8a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:04.606736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.606664 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56htq\" (UniqueName: \"kubernetes.io/projected/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-kube-api-access-56htq\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:04.606736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.606692 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:04.606736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.606702 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:04.606736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.606710 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea8a64a8-e2ff-4427-85b7-8d559346a8a8-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:04.904166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.904129 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:04.907738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.907710 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh"] Oct 30 04:32:04.911428 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:04.911393 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-71b161-84686654cb-vn4gh"] Oct 30 04:32:06.516637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:06.516603 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:06.865541 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:06.865462 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8a64a8-e2ff-4427-85b7-8d559346a8a8" path="/var/lib/kubelet/pods/ea8a64a8-e2ff-4427-85b7-8d559346a8a8/volumes" Oct 30 04:32:07.409756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.409702 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:32:07.443633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.443606 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-socket\") pod \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " Oct 30 04:32:07.443766 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.443643 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-envoy\") pod \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " Oct 30 04:32:07.443766 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.443709 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-credential-socket\") pod \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " Oct 30 04:32:07.443766 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.443760 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istiod-ca-cert\") pod \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " Oct 30 04:32:07.443908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.443831 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-token\" (UniqueName: \"kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-token\") pod \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " Oct 30 04:32:07.443908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.443869 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-kube-api-access-tl4mb\" (UniqueName: \"kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-kube-api-access-tl4mb\") pod \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " Oct 30 04:32:07.443908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.443873 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-socket" pod "a64b9de2-a39e-4d00-8283-1fad481ca2d2" (UID: "a64b9de2-a39e-4d00-8283-1fad481ca2d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:07.444062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.443976 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-credential-socket" pod "a64b9de2-a39e-4d00-8283-1fad481ca2d2" (UID: "a64b9de2-a39e-4d00-8283-1fad481ca2d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:07.444062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.444003 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-data\") pod \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " Oct 30 04:32:07.444062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.444040 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-certs\") pod \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " Oct 30 04:32:07.444229 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.444163 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-podinfo\") pod \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\" (UID: \"a64b9de2-a39e-4d00-8283-1fad481ca2d2\") " Oct 30 04:32:07.444316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.444265 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-data" pod "a64b9de2-a39e-4d00-8283-1fad481ca2d2" (UID: "a64b9de2-a39e-4d00-8283-1fad481ca2d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:07.444316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.444301 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istiod-ca-cert" pod "a64b9de2-a39e-4d00-8283-1fad481ca2d2" (UID: "a64b9de2-a39e-4d00-8283-1fad481ca2d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:07.444453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.444326 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-certs" pod "a64b9de2-a39e-4d00-8283-1fad481ca2d2" (UID: "a64b9de2-a39e-4d00-8283-1fad481ca2d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:07.444738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.444718 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:07.444824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.444742 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:07.444824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.444755 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:07.444824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.444768 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:07.444824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.444780 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:07.446038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.446016 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-envoy" pod "a64b9de2-a39e-4d00-8283-1fad481ca2d2" (UID: "a64b9de2-a39e-4d00-8283-1fad481ca2d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:07.446142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.446033 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-token" pod "a64b9de2-a39e-4d00-8283-1fad481ca2d2" (UID: "a64b9de2-a39e-4d00-8283-1fad481ca2d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:07.446142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.446066 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-kube-api-access-tl4mb" pod "a64b9de2-a39e-4d00-8283-1fad481ca2d2" (UID: "a64b9de2-a39e-4d00-8283-1fad481ca2d2"). InnerVolumeSpecName "kube-api-access-tl4mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:07.447014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.446993 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-podinfo" pod "a64b9de2-a39e-4d00-8283-1fad481ca2d2" (UID: "a64b9de2-a39e-4d00-8283-1fad481ca2d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:32:07.546007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.545893 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:07.546007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.545927 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:07.546007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.545941 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:07.546007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.545955 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl4mb\" (UniqueName: \"kubernetes.io/projected/a64b9de2-a39e-4d00-8283-1fad481ca2d2-kube-api-access-tl4mb\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:07.581348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.581306 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:07.584249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.584221 4793 generic.go:356] "Generic (PLEG): container finished" podID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerID="4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4" exitCode=0 Oct 30 04:32:07.584361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.584291 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" Oct 30 04:32:07.584361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.584299 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" event={"ID":"a64b9de2-a39e-4d00-8283-1fad481ca2d2","Type":"ContainerDied","Data":"4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4"} Oct 30 04:32:07.584361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.584351 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m" event={"ID":"a64b9de2-a39e-4d00-8283-1fad481ca2d2","Type":"ContainerDied","Data":"704c912dc1d13e21c9780ea55952e08c9e5892a75825cfeb7323391a23e00017"} Oct 30 04:32:07.584524 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.584375 4793 scope.go:122] "RemoveContainer" containerID="4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4" Oct 30 04:32:07.597479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.597390 4793 scope.go:122] "RemoveContainer" containerID="a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40" Oct 30 04:32:07.607267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.607252 4793 scope.go:122] "RemoveContainer" containerID="4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4" Oct 30 04:32:07.607537 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:07.607507 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4\": container with ID starting with 4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4 not found: ID does not exist" containerID="4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4" Oct 30 04:32:07.607631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.607547 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4"} err="failed to get container status \"4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4\": rpc error: code = NotFound desc = could not find container \"4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4\": container with ID starting with 4b07d337f8a4664093dcda05d8653751cda39fa4faf02549852e935928c848e4 not found: ID does not exist" Oct 30 04:32:07.607631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.607565 4793 scope.go:122] "RemoveContainer" containerID="a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40" Oct 30 04:32:07.607874 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:07.607852 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40\": container with ID starting with a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40 not found: ID does not exist" containerID="a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40" Oct 30 04:32:07.607919 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.607880 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40"} err="failed to get container status \"a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40\": rpc error: code = NotFound desc = could not find container \"a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40\": container with ID starting with a96c44cdfe504f297ca320cf65788c9fc332d0abb734b62dfd551aa096ef8d40 not found: ID does not exist" Oct 30 04:32:07.640530 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.640505 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m"] Oct 30 04:32:07.644613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.644592 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-58cdc8-openshift-default-846ddb74d5-9t69m"] Oct 30 04:32:07.815752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:07.815717 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:08.030681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:08.030652 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:08.293045 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:08.293008 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:08.500656 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:08.500621 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:08.666505 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:08.655162 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d"] Oct 30 04:32:08.666505 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:08.655729 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" containerID="cri-o://abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf" gracePeriod=30 Oct 30 04:32:08.801151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:08.801096 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:08.853033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:08.852993 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" path="/var/lib/kubelet/pods/a64b9de2-a39e-4d00-8283-1fad481ca2d2/volumes" Oct 30 04:32:08.917607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:08.917535 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4r5r_db99ac60-acfe-44f7-9729-9ef0ef451830/payload-processing/0.log" Oct 30 04:32:09.141948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:09.141923 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4r5r_db99ac60-acfe-44f7-9729-9ef0ef451830/payload-processing/0.log" Oct 30 04:32:09.351735 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:09.351703 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg_859cce50-f6af-42e6-a3d6-9a4c76f04c7f/payload-processing/0.log" Oct 30 04:32:09.647500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:09.634159 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:09.986092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:09.986056 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:10.509011 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:10.508984 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:10.645779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:10.645747 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:32:10.646018 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:10.645804 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:32:11.218853 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:11.218821 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:32:11.863722 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:11.863686 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:13.750134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:13.750092 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:13.990840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:13.990808 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:14.011956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.011928 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:32:14.160804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.160767 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-data\") pod \"62379909-e590-4685-9735-5f3def3f7d99\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " Oct 30 04:32:14.160997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.160844 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-istio-token\" (UniqueName: \"kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-istio-token\") pod \"62379909-e590-4685-9735-5f3def3f7d99\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " Oct 30 04:32:14.160997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.160921 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-socket\") pod \"62379909-e590-4685-9735-5f3def3f7d99\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " Oct 30 04:32:14.161116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.161034 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-certs\") pod \"62379909-e590-4685-9735-5f3def3f7d99\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " Oct 30 04:32:14.161116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.161083 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-data" pod "62379909-e590-4685-9735-5f3def3f7d99" (UID: "62379909-e590-4685-9735-5f3def3f7d99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:14.161116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.161089 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-kube-api-access-q5wbt\" (UniqueName: \"kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-kube-api-access-q5wbt\") pod \"62379909-e590-4685-9735-5f3def3f7d99\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " Oct 30 04:32:14.161277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.161168 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62379909-e590-4685-9735-5f3def3f7d99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62379909-e590-4685-9735-5f3def3f7d99-istiod-ca-cert\") pod \"62379909-e590-4685-9735-5f3def3f7d99\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " Oct 30 04:32:14.161277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.161209 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62379909-e590-4685-9735-5f3def3f7d99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62379909-e590-4685-9735-5f3def3f7d99-istio-podinfo\") pod \"62379909-e590-4685-9735-5f3def3f7d99\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " Oct 30 04:32:14.161277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.161262 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-credential-socket\") pod \"62379909-e590-4685-9735-5f3def3f7d99\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " Oct 30 04:32:14.161480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.161462 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-envoy\") pod \"62379909-e590-4685-9735-5f3def3f7d99\" (UID: \"62379909-e590-4685-9735-5f3def3f7d99\") " Oct 30 04:32:14.161480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.161465 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-certs" pod "62379909-e590-4685-9735-5f3def3f7d99" (UID: "62379909-e590-4685-9735-5f3def3f7d99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:14.162203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.161959 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-credential-socket" pod "62379909-e590-4685-9735-5f3def3f7d99" (UID: "62379909-e590-4685-9735-5f3def3f7d99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:14.162203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.162134 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:14.162203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.162156 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:14.162203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.162171 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:14.162587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.162336 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-socket" pod "62379909-e590-4685-9735-5f3def3f7d99" (UID: "62379909-e590-4685-9735-5f3def3f7d99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:14.162587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.162043 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62379909-e590-4685-9735-5f3def3f7d99-istiod-ca-cert" pod "62379909-e590-4685-9735-5f3def3f7d99" (UID: "62379909-e590-4685-9735-5f3def3f7d99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:14.163993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.163964 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-envoy" pod "62379909-e590-4685-9735-5f3def3f7d99" (UID: "62379909-e590-4685-9735-5f3def3f7d99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:14.164126 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.163997 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62379909-e590-4685-9735-5f3def3f7d99-istio-podinfo" pod "62379909-e590-4685-9735-5f3def3f7d99" (UID: "62379909-e590-4685-9735-5f3def3f7d99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:32:14.164607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.164478 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-istio-token" pod "62379909-e590-4685-9735-5f3def3f7d99" (UID: "62379909-e590-4685-9735-5f3def3f7d99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:14.164607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.164576 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-kube-api-access-q5wbt" pod "62379909-e590-4685-9735-5f3def3f7d99" (UID: "62379909-e590-4685-9735-5f3def3f7d99"). InnerVolumeSpecName "kube-api-access-q5wbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:14.190587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.190550 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:14.263392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.263331 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:14.263392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.263360 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:14.263392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.263375 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5wbt\" (UniqueName: \"kubernetes.io/projected/62379909-e590-4685-9735-5f3def3f7d99-kube-api-access-q5wbt\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:14.263392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.263391 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62379909-e590-4685-9735-5f3def3f7d99-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:14.263615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.263405 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62379909-e590-4685-9735-5f3def3f7d99-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:14.263615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.263433 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62379909-e590-4685-9735-5f3def3f7d99-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:14.633257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.633162 4793 generic.go:356] "Generic (PLEG): container finished" podID="62379909-e590-4685-9735-5f3def3f7d99" containerID="abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf" exitCode=0 Oct 30 04:32:14.633257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.633218 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" Oct 30 04:32:14.633257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.633237 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" event={"ID":"62379909-e590-4685-9735-5f3def3f7d99","Type":"ContainerDied","Data":"abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf"} Oct 30 04:32:14.633536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.633291 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d" event={"ID":"62379909-e590-4685-9735-5f3def3f7d99","Type":"ContainerDied","Data":"4514349305b722e14f812266f97f2a65d450843cde88226838e0acab217620ba"} Oct 30 04:32:14.633536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.633316 4793 scope.go:122] "RemoveContainer" containerID="abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf" Oct 30 04:32:14.646022 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.645999 4793 scope.go:122] "RemoveContainer" containerID="701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498" Oct 30 04:32:14.668358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.668336 4793 scope.go:122] "RemoveContainer" containerID="abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf" Oct 30 04:32:14.668697 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:14.668673 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf\": container with ID starting with abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf not found: ID does not exist" containerID="abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf" Oct 30 04:32:14.668807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.668704 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf"} err="failed to get container status \"abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf\": rpc error: code = NotFound desc = could not find container \"abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf\": container with ID starting with abd740a9695f9d340da25912f2065bfe09b0b15bcb8859da8b85fb835605c4cf not found: ID does not exist" Oct 30 04:32:14.668807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.668725 4793 scope.go:122] "RemoveContainer" containerID="701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498" Oct 30 04:32:14.669047 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:14.669021 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498\": container with ID starting with 701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498 not found: ID does not exist" containerID="701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498" Oct 30 04:32:14.669093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.669059 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498"} err="failed to get container status \"701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498\": rpc error: code = NotFound desc = could not find container \"701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498\": container with ID starting with 701cc375806c36001c60ad5807b952e029923be3ddf89bbe316c6bfdacf38498 not found: ID does not exist" Oct 30 04:32:14.709064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.709039 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:14.710942 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.710916 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d"] Oct 30 04:32:14.723045 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.723021 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-71b161-openshift-default-59949694fc-x4c8d"] Oct 30 04:32:14.847787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.847753 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62379909-e590-4685-9735-5f3def3f7d99" path="/var/lib/kubelet/pods/62379909-e590-4685-9735-5f3def3f7d99/volumes" Oct 30 04:32:14.926406 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:14.926377 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:15.126602 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:15.126560 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:15.740646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:15.740614 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62"] Oct 30 04:32:15.740952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:15.740893 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" podUID="fb8b5dc2-4bc4-4580-b82a-747d9ce48611" containerName="maas-api" containerID="cri-o://63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3" gracePeriod=30 Oct 30 04:32:15.896364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:15.896305 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:16.077480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.077452 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:32:16.192343 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.192312 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-kube-api-access-99wn6\" (UniqueName: \"kubernetes.io/projected/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-kube-api-access-99wn6\") pod \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " Oct 30 04:32:16.192534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.192399 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-metrics-certs\") pod \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " Oct 30 04:32:16.192621 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.192596 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-openshift-service-ca\") pod \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " Oct 30 04:32:16.192680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.192634 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-maas-api-tls\") pod \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\" (UID: \"fb8b5dc2-4bc4-4580-b82a-747d9ce48611\") " Oct 30 04:32:16.193007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.192947 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-openshift-service-ca" pod "fb8b5dc2-4bc4-4580-b82a-747d9ce48611" (UID: "fb8b5dc2-4bc4-4580-b82a-747d9ce48611"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:16.193229 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.193210 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:16.194770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.194737 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-kube-api-access-99wn6" pod "fb8b5dc2-4bc4-4580-b82a-747d9ce48611" (UID: "fb8b5dc2-4bc4-4580-b82a-747d9ce48611"). InnerVolumeSpecName "kube-api-access-99wn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:16.194971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.194944 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-metrics-certs" pod "fb8b5dc2-4bc4-4580-b82a-747d9ce48611" (UID: "fb8b5dc2-4bc4-4580-b82a-747d9ce48611"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:16.195174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.195158 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-maas-api-tls" pod "fb8b5dc2-4bc4-4580-b82a-747d9ce48611" (UID: "fb8b5dc2-4bc4-4580-b82a-747d9ce48611"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:16.297564 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.297470 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:16.297782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.297766 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:16.297877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.297864 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99wn6\" (UniqueName: \"kubernetes.io/projected/fb8b5dc2-4bc4-4580-b82a-747d9ce48611-kube-api-access-99wn6\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:16.646684 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.646604 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:16.648811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.648785 4793 generic.go:356] "Generic (PLEG): container finished" podID="fb8b5dc2-4bc4-4580-b82a-747d9ce48611" containerID="63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3" exitCode=0 Oct 30 04:32:16.648923 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.648837 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" Oct 30 04:32:16.648923 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.648875 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" event={"ID":"fb8b5dc2-4bc4-4580-b82a-747d9ce48611","Type":"ContainerDied","Data":"63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3"} Oct 30 04:32:16.649005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.648933 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62" event={"ID":"fb8b5dc2-4bc4-4580-b82a-747d9ce48611","Type":"ContainerDied","Data":"8880691aab83b82632653afa53cadc14ba93abda83a71cd91fe0e9e3d607b547"} Oct 30 04:32:16.649005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.648956 4793 scope.go:122] "RemoveContainer" containerID="63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3" Oct 30 04:32:16.659459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.659436 4793 scope.go:122] "RemoveContainer" containerID="63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3" Oct 30 04:32:16.659784 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:16.659764 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3\": container with ID starting with 63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3 not found: ID does not exist" containerID="63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3" Oct 30 04:32:16.659869 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.659795 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3"} err="failed to get container status \"63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3\": rpc error: code = NotFound desc = could not find container \"63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3\": container with ID starting with 63c6f0bcc81f01fa25a9b9b9e318e9cba4dca488838b75096463b1e90733a8e3 not found: ID does not exist" Oct 30 04:32:16.707171 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.707145 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62"] Oct 30 04:32:16.714978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.714950 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-05f827-66656c59d9-rjw62"] Oct 30 04:32:16.846537 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.844403 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:16.846930 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:16.846907 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb8b5dc2-4bc4-4580-b82a-747d9ce48611" path="/var/lib/kubelet/pods/fb8b5dc2-4bc4-4580-b82a-747d9ce48611/volumes" Oct 30 04:32:17.615189 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.615153 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948"] Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.615996 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ea8a64a8-e2ff-4427-85b7-8d559346a8a8" containerName="maas-api" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616019 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8a64a8-e2ff-4427-85b7-8d559346a8a8" containerName="maas-api" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616051 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerName="istio-proxy" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616058 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerName="istio-proxy" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616072 4793 cpu_manager.go:414] "containerMap: removing container" podUID="fb8b5dc2-4bc4-4580-b82a-747d9ce48611" containerName="maas-api" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616079 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb8b5dc2-4bc4-4580-b82a-747d9ce48611" containerName="maas-api" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616095 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerName="istio-proxy" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616101 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerName="istio-proxy" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616114 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ccc2c258-ffe4-4e16-bf5e-996d41b90f45" containerName="maas-api" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616120 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc2c258-ffe4-4e16-bf5e-996d41b90f45" containerName="maas-api" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616151 4793 cpu_manager.go:414] "containerMap: removing container" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616159 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616168 4793 cpu_manager.go:414] "containerMap: removing container" podUID="23e14557-01b4-4800-980a-0ab377967fcf" containerName="payload-processing" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616174 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23e14557-01b4-4800-980a-0ab377967fcf" containerName="payload-processing" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616196 4793 cpu_manager.go:414] "containerMap: removing container" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" Oct 30 04:32:17.616257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616202 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" Oct 30 04:32:17.617026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616332 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8a64a8-e2ff-4427-85b7-8d559346a8a8" containerName="maas-api" Oct 30 04:32:17.617026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616347 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerName="istio-proxy" Oct 30 04:32:17.617026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616356 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb8b5dc2-4bc4-4580-b82a-747d9ce48611" containerName="maas-api" Oct 30 04:32:17.617026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616366 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc2c258-ffe4-4e16-bf5e-996d41b90f45" containerName="maas-api" Oct 30 04:32:17.617026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616379 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" Oct 30 04:32:17.617026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616387 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="23e14557-01b4-4800-980a-0ab377967fcf" containerName="payload-processing" Oct 30 04:32:17.617026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.616399 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="62379909-e590-4685-9735-5f3def3f7d99" containerName="istio-proxy" Oct 30 04:32:17.626117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.626091 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948"] Oct 30 04:32:17.626222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.626199 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.655458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.654761 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-153a44\"/\"openshift-service-ca.crt\"" Oct 30 04:32:17.655458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.654865 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-153a44\"/\"default-dockercfg-8ffx9\"" Oct 30 04:32:17.655458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.654937 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-153a44\"/\"kube-root-ca.crt\"" Oct 30 04:32:17.655458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.655071 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-153a44\"/\"praxis-route-153a44-kserve-self-signed-certs\"" Oct 30 04:32:17.696955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.696926 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:17.718931 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.718903 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-dshm\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.719102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.718935 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-model-cache\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.719102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.718978 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-tmp-dir\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.719206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.719097 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rmq7\" (UniqueName: \"kubernetes.io/projected/6cc03e55-64ec-44bb-934f-6555acc3c342-kube-api-access-6rmq7\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.719206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.719191 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cc03e55-64ec-44bb-934f-6555acc3c342-tls-certs\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.719309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.719225 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-home\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.832442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.822577 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-dshm\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.832739 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.832707 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-model-cache\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.832939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.832920 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-tmp-dir\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.833112 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.833092 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rmq7\" (UniqueName: \"kubernetes.io/projected/6cc03e55-64ec-44bb-934f-6555acc3c342-kube-api-access-6rmq7\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.833307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.833291 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cc03e55-64ec-44bb-934f-6555acc3c342-tls-certs\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.833499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.833483 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-home\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.839450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.834043 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-home\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.845842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.839918 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-tmp-dir\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.846180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.840182 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-model-cache\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.848885 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.848862 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cc03e55-64ec-44bb-934f-6555acc3c342-tls-certs\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.853441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.849663 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-dshm\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.868966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.865953 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rmq7\" (UniqueName: \"kubernetes.io/projected/6cc03e55-64ec-44bb-934f-6555acc3c342-kube-api-access-6rmq7\") pod \"praxis-route-153a44-kserve-68fc94df89-zd948\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:17.898022 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.897865 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:17.941601 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:17.941568 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:18.248852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:18.248823 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948"] Oct 30 04:32:18.249018 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:32:18.248997 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cc03e55_64ec_44bb_934f_6555acc3c342.slice/crio-cd6be9e72189bc45d6706b4d9f25556b59d031b6f3c3b8533b82c4aa30468142 WatchSource:0}: Error finding container cd6be9e72189bc45d6706b4d9f25556b59d031b6f3c3b8533b82c4aa30468142: Status 404 returned error can't find the container with id cd6be9e72189bc45d6706b4d9f25556b59d031b6f3c3b8533b82c4aa30468142 Oct 30 04:32:18.677634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:18.677595 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" event={"ID":"6cc03e55-64ec-44bb-934f-6555acc3c342","Type":"ContainerStarted","Data":"8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1"} Oct 30 04:32:18.678150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:18.677644 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" event={"ID":"6cc03e55-64ec-44bb-934f-6555acc3c342","Type":"ContainerStarted","Data":"cd6be9e72189bc45d6706b4d9f25556b59d031b6f3c3b8533b82c4aa30468142"} Oct 30 04:32:18.723631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:18.723584 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" podStartSLOduration=1.723569347 podStartE2EDuration="1.723569347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:32:18.720592775 +0000 UTC m=+2062.006467550" watchObservedRunningTime="2026-10-30 04:32:18.723569347 +0000 UTC m=+2062.009444123" Oct 30 04:32:18.941731 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:18.941653 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:19.126461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:19.115027 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:20.856764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:20.856725 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:21.874105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:21.874069 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:22.229821 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:22.229785 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:22.429636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:22.429600 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:22.635091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:22.635014 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:22.878479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:22.878439 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:23.162700 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:23.162663 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp"] Oct 30 04:32:23.162954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:23.162926 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" containerID="cri-o://156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730" gracePeriod=30 Oct 30 04:32:23.300442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:23.299503 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:24.922334 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:24.922294 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:27.070218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:27.070180 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:27.942095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:27.942065 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:27.954489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:27.954462 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:28.441647 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.441623 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:32:28.590932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.590849 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-kube-api-access-fpvdj\" (UniqueName: \"kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-kube-api-access-fpvdj\") pod \"b31e3220-e28e-432d-9f9a-86ce56e66a78\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " Oct 30 04:32:28.590932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.590903 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-podinfo\") pod \"b31e3220-e28e-432d-9f9a-86ce56e66a78\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " Oct 30 04:32:28.591159 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.591000 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-data\") pod \"b31e3220-e28e-432d-9f9a-86ce56e66a78\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " Oct 30 04:32:28.591159 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.591027 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-token\" (UniqueName: \"kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-token\") pod \"b31e3220-e28e-432d-9f9a-86ce56e66a78\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " Oct 30 04:32:28.591159 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.591089 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-envoy\") pod \"b31e3220-e28e-432d-9f9a-86ce56e66a78\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " Oct 30 04:32:28.591365 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.591337 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-data" pod "b31e3220-e28e-432d-9f9a-86ce56e66a78" (UID: "b31e3220-e28e-432d-9f9a-86ce56e66a78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:28.591537 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.591509 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-certs\") pod \"b31e3220-e28e-432d-9f9a-86ce56e66a78\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " Oct 30 04:32:28.591624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.591570 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-socket\") pod \"b31e3220-e28e-432d-9f9a-86ce56e66a78\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " Oct 30 04:32:28.591688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.591626 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31e3220-e28e-432d-9f9a-86ce56e66a78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31e3220-e28e-432d-9f9a-86ce56e66a78-istiod-ca-cert\") pod \"b31e3220-e28e-432d-9f9a-86ce56e66a78\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " Oct 30 04:32:28.591688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.591658 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-credential-socket\") pod \"b31e3220-e28e-432d-9f9a-86ce56e66a78\" (UID: \"b31e3220-e28e-432d-9f9a-86ce56e66a78\") " Oct 30 04:32:28.592126 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.592077 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-socket" pod "b31e3220-e28e-432d-9f9a-86ce56e66a78" (UID: "b31e3220-e28e-432d-9f9a-86ce56e66a78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:28.592337 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.592300 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-credential-socket" pod "b31e3220-e28e-432d-9f9a-86ce56e66a78" (UID: "b31e3220-e28e-432d-9f9a-86ce56e66a78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:28.592441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.592370 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-certs" pod "b31e3220-e28e-432d-9f9a-86ce56e66a78" (UID: "b31e3220-e28e-432d-9f9a-86ce56e66a78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:28.592561 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.592539 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31e3220-e28e-432d-9f9a-86ce56e66a78-istiod-ca-cert" pod "b31e3220-e28e-432d-9f9a-86ce56e66a78" (UID: "b31e3220-e28e-432d-9f9a-86ce56e66a78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:28.593009 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.592988 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:28.593103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.593016 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:28.593103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.593032 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b31e3220-e28e-432d-9f9a-86ce56e66a78-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:28.593103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.593047 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:28.593103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.593061 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:28.593716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.593689 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-token" pod "b31e3220-e28e-432d-9f9a-86ce56e66a78" (UID: "b31e3220-e28e-432d-9f9a-86ce56e66a78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:28.593884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.593751 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-envoy" pod "b31e3220-e28e-432d-9f9a-86ce56e66a78" (UID: "b31e3220-e28e-432d-9f9a-86ce56e66a78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:28.593884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.593753 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-podinfo" pod "b31e3220-e28e-432d-9f9a-86ce56e66a78" (UID: "b31e3220-e28e-432d-9f9a-86ce56e66a78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:32:28.594009 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.593985 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-kube-api-access-fpvdj" pod "b31e3220-e28e-432d-9f9a-86ce56e66a78" (UID: "b31e3220-e28e-432d-9f9a-86ce56e66a78"). InnerVolumeSpecName "kube-api-access-fpvdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:28.699764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.694303 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:28.699764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.694348 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpvdj\" (UniqueName: \"kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-kube-api-access-fpvdj\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:28.699764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.694364 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:28.699764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.694381 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b31e3220-e28e-432d-9f9a-86ce56e66a78-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:28.737232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.737195 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:28.751975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.751945 4793 generic.go:356] "Generic (PLEG): container finished" podID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerID="156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730" exitCode=0 Oct 30 04:32:28.752102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.752014 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" Oct 30 04:32:28.752102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.752032 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" event={"ID":"b31e3220-e28e-432d-9f9a-86ce56e66a78","Type":"ContainerDied","Data":"156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730"} Oct 30 04:32:28.752102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.752083 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp" event={"ID":"b31e3220-e28e-432d-9f9a-86ce56e66a78","Type":"ContainerDied","Data":"3e04dfcf68e1a70892103116e8757b15a1856ae409a9dfed47e6d77a376c5ff7"} Oct 30 04:32:28.752245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.752104 4793 scope.go:122] "RemoveContainer" containerID="156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730" Oct 30 04:32:28.764201 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.764181 4793 scope.go:122] "RemoveContainer" containerID="683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592" Oct 30 04:32:28.768501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.768479 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:32:28.775683 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.775666 4793 scope.go:122] "RemoveContainer" containerID="156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730" Oct 30 04:32:28.775953 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:28.775935 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730\": container with ID starting with 156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730 not found: ID does not exist" containerID="156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730" Oct 30 04:32:28.776025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.775960 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730"} err="failed to get container status \"156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730\": rpc error: code = NotFound desc = could not find container \"156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730\": container with ID starting with 156920dbcfcca930b66ebb808af2705433ba8738f617e34896a6e282d6fd4730 not found: ID does not exist" Oct 30 04:32:28.776025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.775977 4793 scope.go:122] "RemoveContainer" containerID="683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592" Oct 30 04:32:28.776242 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:28.776218 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592\": container with ID starting with 683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592 not found: ID does not exist" containerID="683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592" Oct 30 04:32:28.776292 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.776254 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592"} err="failed to get container status \"683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592\": rpc error: code = NotFound desc = could not find container \"683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592\": container with ID starting with 683c918a362f7bbf13505f4f9f4b6f8c1eaea24f04f413d962be67996d4dd592 not found: ID does not exist" Oct 30 04:32:28.808101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.808081 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp"] Oct 30 04:32:28.819138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.819077 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-05f827-openshift-default-6589fdcf57-5sncp"] Oct 30 04:32:28.846164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:28.846106 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" path="/var/lib/kubelet/pods/b31e3220-e28e-432d-9f9a-86ce56e66a78/volumes" Oct 30 04:32:30.395927 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:30.395883 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:31.766920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:31.766874 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp"] Oct 30 04:32:31.767592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:31.767562 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" podUID="e8c1fe84-875c-4d1b-b083-6a368cbded98" containerName="maas-api" containerID="cri-o://5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9" gracePeriod=30 Oct 30 04:32:31.940409 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:31.940373 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:32.064519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.061058 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:32:32.142405 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.142373 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8c1fe84-875c-4d1b-b083-6a368cbded98-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c1fe84-875c-4d1b-b083-6a368cbded98-openshift-service-ca\") pod \"e8c1fe84-875c-4d1b-b083-6a368cbded98\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " Oct 30 04:32:32.142629 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.142595 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-metrics-certs\") pod \"e8c1fe84-875c-4d1b-b083-6a368cbded98\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " Oct 30 04:32:32.142758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.142646 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c1fe84-875c-4d1b-b083-6a368cbded98-kube-api-access-tmn6v\" (UniqueName: \"kubernetes.io/projected/e8c1fe84-875c-4d1b-b083-6a368cbded98-kube-api-access-tmn6v\") pod \"e8c1fe84-875c-4d1b-b083-6a368cbded98\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " Oct 30 04:32:32.142758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.142706 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-maas-api-tls\") pod \"e8c1fe84-875c-4d1b-b083-6a368cbded98\" (UID: \"e8c1fe84-875c-4d1b-b083-6a368cbded98\") " Oct 30 04:32:32.143366 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.143336 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c1fe84-875c-4d1b-b083-6a368cbded98-openshift-service-ca" pod "e8c1fe84-875c-4d1b-b083-6a368cbded98" (UID: "e8c1fe84-875c-4d1b-b083-6a368cbded98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:32.146232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.146170 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-metrics-certs" pod "e8c1fe84-875c-4d1b-b083-6a368cbded98" (UID: "e8c1fe84-875c-4d1b-b083-6a368cbded98"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:32.146232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.146211 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c1fe84-875c-4d1b-b083-6a368cbded98-kube-api-access-tmn6v" pod "e8c1fe84-875c-4d1b-b083-6a368cbded98" (UID: "e8c1fe84-875c-4d1b-b083-6a368cbded98"). InnerVolumeSpecName "kube-api-access-tmn6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:32.146650 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.146611 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-maas-api-tls" pod "e8c1fe84-875c-4d1b-b083-6a368cbded98" (UID: "e8c1fe84-875c-4d1b-b083-6a368cbded98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:32.264181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.264139 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:32.264778 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.264751 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8c1fe84-875c-4d1b-b083-6a368cbded98-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:32.264937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.264922 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8c1fe84-875c-4d1b-b083-6a368cbded98-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:32.265049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.265037 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmn6v\" (UniqueName: \"kubernetes.io/projected/e8c1fe84-875c-4d1b-b083-6a368cbded98-kube-api-access-tmn6v\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:32.643758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.643721 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:32.701070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.701040 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5"] Oct 30 04:32:32.701607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.701592 4793 cpu_manager.go:414] "containerMap: removing container" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" Oct 30 04:32:32.701607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.701607 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" Oct 30 04:32:32.701716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.701628 4793 cpu_manager.go:414] "containerMap: removing container" podUID="e8c1fe84-875c-4d1b-b083-6a368cbded98" containerName="maas-api" Oct 30 04:32:32.701716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.701635 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8c1fe84-875c-4d1b-b083-6a368cbded98" containerName="maas-api" Oct 30 04:32:32.701781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.701726 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8c1fe84-875c-4d1b-b083-6a368cbded98" containerName="maas-api" Oct 30 04:32:32.701781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.701738 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" Oct 30 04:32:32.701781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.701746 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a64b9de2-a39e-4d00-8283-1fad481ca2d2" containerName="istio-proxy" Oct 30 04:32:32.701781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.701753 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" Oct 30 04:32:32.705300 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.705263 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:32.720524 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.720501 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:32:32.722700 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.722681 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5"] Oct 30 04:32:32.731285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.731260 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:32:32.770141 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.770112 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef8543e5-jvtb5\" (UID: \"d338b3ca-77cc-4e7d-a1f4-202833a479df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:32.770666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.770251 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef8543e5-jvtb5\" (UID: \"d338b3ca-77cc-4e7d-a1f4-202833a479df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:32.770666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.770385 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqw7f\" (UniqueName: \"kubernetes.io/projected/d338b3ca-77cc-4e7d-a1f4-202833a479df-kube-api-access-xqw7f\") pod \"maas-api-revoke-sub-ef8543e5-jvtb5\" (UID: \"d338b3ca-77cc-4e7d-a1f4-202833a479df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:32.806579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.806544 4793 generic.go:356] "Generic (PLEG): container finished" podID="e8c1fe84-875c-4d1b-b083-6a368cbded98" containerID="5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9" exitCode=0 Oct 30 04:32:32.806694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.806613 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" Oct 30 04:32:32.806694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.806623 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" event={"ID":"e8c1fe84-875c-4d1b-b083-6a368cbded98","Type":"ContainerDied","Data":"5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9"} Oct 30 04:32:32.806769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.806695 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp" event={"ID":"e8c1fe84-875c-4d1b-b083-6a368cbded98","Type":"ContainerDied","Data":"6edb0357a86c712645813e36424d254fd76506ba695dd556ddce98f581534081"} Oct 30 04:32:32.806769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.806717 4793 scope.go:122] "RemoveContainer" containerID="5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9" Oct 30 04:32:32.817146 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.817123 4793 scope.go:122] "RemoveContainer" containerID="5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9" Oct 30 04:32:32.817401 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:32.817381 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9\": container with ID starting with 5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9 not found: ID does not exist" containerID="5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9" Oct 30 04:32:32.817511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.817412 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9"} err="failed to get container status \"5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9\": rpc error: code = NotFound desc = could not find container \"5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9\": container with ID starting with 5389172724b0caeed854b2fee4631a6f4ea13eca9bc1a22dd36395e976e35de9 not found: ID does not exist" Oct 30 04:32:32.862250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.862190 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp"] Oct 30 04:32:32.865194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.865172 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-eb9d23-78bbcf9754-qkwbp"] Oct 30 04:32:32.871508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.871487 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqw7f\" (UniqueName: \"kubernetes.io/projected/d338b3ca-77cc-4e7d-a1f4-202833a479df-kube-api-access-xqw7f\") pod \"maas-api-revoke-sub-ef8543e5-jvtb5\" (UID: \"d338b3ca-77cc-4e7d-a1f4-202833a479df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:32.871613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.871595 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef8543e5-jvtb5\" (UID: \"d338b3ca-77cc-4e7d-a1f4-202833a479df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:32.871685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.871658 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef8543e5-jvtb5\" (UID: \"d338b3ca-77cc-4e7d-a1f4-202833a479df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:32.872311 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.872290 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef8543e5-jvtb5\" (UID: \"d338b3ca-77cc-4e7d-a1f4-202833a479df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:32.874047 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.874023 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef8543e5-jvtb5\" (UID: \"d338b3ca-77cc-4e7d-a1f4-202833a479df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:32.895113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:32.895060 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqw7f\" (UniqueName: \"kubernetes.io/projected/d338b3ca-77cc-4e7d-a1f4-202833a479df-kube-api-access-xqw7f\") pod \"maas-api-revoke-sub-ef8543e5-jvtb5\" (UID: \"d338b3ca-77cc-4e7d-a1f4-202833a479df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:33.019311 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:33.019283 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:33.327293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:33.327264 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5"] Oct 30 04:32:33.328325 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:32:33.328287 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd338b3ca_77cc_4e7d_a1f4_202833a479df.slice/crio-ef156be493701e3798692518b25125d6d6c4ed8a66964672e39816f24acca079 WatchSource:0}: Error finding container ef156be493701e3798692518b25125d6d6c4ed8a66964672e39816f24acca079: Status 404 returned error can't find the container with id ef156be493701e3798692518b25125d6d6c4ed8a66964672e39816f24acca079 Oct 30 04:32:33.817307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:33.817271 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" event={"ID":"d338b3ca-77cc-4e7d-a1f4-202833a479df","Type":"ContainerStarted","Data":"da6a5b2b67d04caf33cb1f893d1eea96f406b30c51a3bf064634c6227491d061"} Oct 30 04:32:33.817691 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:33.817329 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" event={"ID":"d338b3ca-77cc-4e7d-a1f4-202833a479df","Type":"ContainerStarted","Data":"ef156be493701e3798692518b25125d6d6c4ed8a66964672e39816f24acca079"} Oct 30 04:32:33.859726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:33.859683 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" podStartSLOduration=1.8596694280000001 podStartE2EDuration="1.859669428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:32:33.858095249 +0000 UTC m=+2077.143970025" watchObservedRunningTime="2026-10-30 04:32:33.859669428 +0000 UTC m=+2077.145544202" Oct 30 04:32:34.396972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:34.396935 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:34.826153 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:34.826121 4793 generic.go:356] "Generic (PLEG): container finished" podID="d338b3ca-77cc-4e7d-a1f4-202833a479df" containerID="da6a5b2b67d04caf33cb1f893d1eea96f406b30c51a3bf064634c6227491d061" exitCode=0 Oct 30 04:32:34.826547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:34.826215 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" event={"ID":"d338b3ca-77cc-4e7d-a1f4-202833a479df","Type":"ContainerDied","Data":"da6a5b2b67d04caf33cb1f893d1eea96f406b30c51a3bf064634c6227491d061"} Oct 30 04:32:34.846168 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:34.846146 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8c1fe84-875c-4d1b-b083-6a368cbded98" path="/var/lib/kubelet/pods/e8c1fe84-875c-4d1b-b083-6a368cbded98/volumes" Oct 30 04:32:36.001504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.001470 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:36.106134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.106034 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:36.110123 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.110102 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d338b3ca-77cc-4e7d-a1f4-202833a479df-kube-api-access-xqw7f\" (UniqueName: \"kubernetes.io/projected/d338b3ca-77cc-4e7d-a1f4-202833a479df-kube-api-access-xqw7f\") pod \"d338b3ca-77cc-4e7d-a1f4-202833a479df\" (UID: \"d338b3ca-77cc-4e7d-a1f4-202833a479df\") " Oct 30 04:32:36.110264 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.110169 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-xks-ca\") pod \"d338b3ca-77cc-4e7d-a1f4-202833a479df\" (UID: \"d338b3ca-77cc-4e7d-a1f4-202833a479df\") " Oct 30 04:32:36.110264 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.110205 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-openshift-ca\") pod \"d338b3ca-77cc-4e7d-a1f4-202833a479df\" (UID: \"d338b3ca-77cc-4e7d-a1f4-202833a479df\") " Oct 30 04:32:36.110605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.110577 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-openshift-ca" pod "d338b3ca-77cc-4e7d-a1f4-202833a479df" (UID: "d338b3ca-77cc-4e7d-a1f4-202833a479df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:36.110835 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.110816 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:36.112484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.112457 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-xks-ca" pod "d338b3ca-77cc-4e7d-a1f4-202833a479df" (UID: "d338b3ca-77cc-4e7d-a1f4-202833a479df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:36.113614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.113592 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d338b3ca-77cc-4e7d-a1f4-202833a479df-kube-api-access-xqw7f" pod "d338b3ca-77cc-4e7d-a1f4-202833a479df" (UID: "d338b3ca-77cc-4e7d-a1f4-202833a479df"). InnerVolumeSpecName "kube-api-access-xqw7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:36.212383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.212353 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqw7f\" (UniqueName: \"kubernetes.io/projected/d338b3ca-77cc-4e7d-a1f4-202833a479df-kube-api-access-xqw7f\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:36.212383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.212379 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d338b3ca-77cc-4e7d-a1f4-202833a479df-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:36.358643 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.358563 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb_aa8a7afc-f51a-49ce-8837-0d5f39b03600/payload-processing/0.log" Oct 30 04:32:36.592464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.592435 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4r5r_db99ac60-acfe-44f7-9729-9ef0ef451830/payload-processing/0.log" Oct 30 04:32:36.817901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.817876 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-l4r5r_db99ac60-acfe-44f7-9729-9ef0ef451830/payload-processing/0.log" Oct 30 04:32:36.949446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.945760 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" Oct 30 04:32:36.952698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.952662 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5" event={"ID":"d338b3ca-77cc-4e7d-a1f4-202833a479df","Type":"ContainerDied","Data":"ef156be493701e3798692518b25125d6d6c4ed8a66964672e39816f24acca079"} Oct 30 04:32:36.952698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:36.952722 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef156be493701e3798692518b25125d6d6c4ed8a66964672e39816f24acca079" Oct 30 04:32:37.373908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:37.373866 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:37.629768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:37.629684 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:37.836663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:37.836627 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:38.031000 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:38.030966 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:38.236994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:38.236957 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:38.336146 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:38.336062 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7"] Oct 30 04:32:38.336709 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:38.336676 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" containerID="cri-o://ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3" gracePeriod=30 Oct 30 04:32:38.426778 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:38.426735 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:32:38.427178 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:38.426793 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:32:38.452573 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:38.452543 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:38.658534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:38.658446 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:40.722192 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:40.722163 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:41.221037 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:41.221000 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:32:41.952432 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:41.952381 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:42.157458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:42.157410 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:42.370594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:42.370490 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:42.645586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:42.645498 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:42.932655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:42.926231 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf"] Oct 30 04:32:42.932655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:42.926503 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" containerID="cri-o://f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336" gracePeriod=30 Oct 30 04:32:43.101692 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.101655 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:43.420279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.420248 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:43.621270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.621247 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:32:43.725259 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.725226 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-token\" (UniqueName: \"kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-token\") pod \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " Oct 30 04:32:43.725501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.725283 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-certs\") pod \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " Oct 30 04:32:43.725501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.725304 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istiod-ca-cert\") pod \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " Oct 30 04:32:43.725501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.725342 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-kube-api-access-jg46w\" (UniqueName: \"kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-kube-api-access-jg46w\") pod \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " Oct 30 04:32:43.725501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.725413 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-podinfo\") pod \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " Oct 30 04:32:43.725501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.725483 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-credential-socket\") pod \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " Oct 30 04:32:43.725766 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.725560 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-data\") pod \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " Oct 30 04:32:43.725766 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.725603 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-envoy\") pod \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " Oct 30 04:32:43.725766 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.725624 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-certs" pod "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" (UID: "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:43.725766 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.725723 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-socket\") pod \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\" (UID: \"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a\") " Oct 30 04:32:43.725983 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.725850 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-credential-socket" pod "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" (UID: "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:43.725983 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.725873 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istiod-ca-cert" pod "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" (UID: "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:43.726101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.726074 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-socket" pod "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" (UID: "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:43.726198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.726146 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-data" pod "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" (UID: "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:43.726398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.726373 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:43.726398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.726399 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:43.726587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.726440 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:43.726587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.726453 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:43.726587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.726466 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:43.727888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.727866 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-envoy" pod "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" (UID: "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:43.727999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.727897 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-token" pod "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" (UID: "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:43.728116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.728095 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-kube-api-access-jg46w" pod "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" (UID: "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a"). InnerVolumeSpecName "kube-api-access-jg46w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:43.728176 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.728096 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-podinfo" pod "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" (UID: "7805bda5-d2a2-4d8c-87a9-306ffa87fa4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:32:43.827194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.827168 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:43.827194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.827195 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg46w\" (UniqueName: \"kubernetes.io/projected/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-kube-api-access-jg46w\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:43.827367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.827206 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:43.827367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.827214 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:43.898880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.898799 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:43.998086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.998047 4793 generic.go:356] "Generic (PLEG): container finished" podID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerID="ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3" exitCode=0 Oct 30 04:32:43.998273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.998095 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" Oct 30 04:32:43.998273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.998091 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" event={"ID":"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a","Type":"ContainerDied","Data":"ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3"} Oct 30 04:32:43.998273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.998214 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7" event={"ID":"7805bda5-d2a2-4d8c-87a9-306ffa87fa4a","Type":"ContainerDied","Data":"c22b90fe24b8725370a4dad1b625f3784877307e1604f9102bac790a55a5b5de"} Oct 30 04:32:43.998273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:43.998245 4793 scope.go:122] "RemoveContainer" containerID="ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3" Oct 30 04:32:44.010164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:44.010144 4793 scope.go:122] "RemoveContainer" containerID="7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50" Oct 30 04:32:44.020124 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:44.020103 4793 scope.go:122] "RemoveContainer" containerID="ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3" Oct 30 04:32:44.020393 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:44.020369 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3\": container with ID starting with ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3 not found: ID does not exist" containerID="ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3" Oct 30 04:32:44.020485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:44.020405 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3"} err="failed to get container status \"ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3\": rpc error: code = NotFound desc = could not find container \"ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3\": container with ID starting with ebabf46a97b0007e375f8cb6e02dc4f8e7a1eaa17f229295024b0aa37cb34be3 not found: ID does not exist" Oct 30 04:32:44.020485 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:44.020453 4793 scope.go:122] "RemoveContainer" containerID="7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50" Oct 30 04:32:44.020719 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:44.020698 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50\": container with ID starting with 7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50 not found: ID does not exist" containerID="7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50" Oct 30 04:32:44.020759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:44.020725 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50"} err="failed to get container status \"7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50\": rpc error: code = NotFound desc = could not find container \"7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50\": container with ID starting with 7a73b125d472b3bc71dec2cef10048238947d68e528fec40d154f97a2c3e0f50 not found: ID does not exist" Oct 30 04:32:44.054487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:44.054464 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7"] Oct 30 04:32:44.058735 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:44.058715 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-eb9d23-openshift-default-76b8d78d5c-ch5l7"] Oct 30 04:32:44.292072 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:44.292036 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:44.846619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:44.846591 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" path="/var/lib/kubelet/pods/7805bda5-d2a2-4d8c-87a9-306ffa87fa4a/volumes" Oct 30 04:32:44.854409 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:44.854372 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:32:44.854557 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:44.854439 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:32:45.315385 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.315352 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl"] Oct 30 04:32:45.315921 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.315905 4793 cpu_manager.go:414] "containerMap: removing container" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" Oct 30 04:32:45.315972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.315922 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" Oct 30 04:32:45.315972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.315938 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d338b3ca-77cc-4e7d-a1f4-202833a479df" containerName="revoke-keys" Oct 30 04:32:45.315972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.315944 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d338b3ca-77cc-4e7d-a1f4-202833a479df" containerName="revoke-keys" Oct 30 04:32:45.315972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.315964 4793 cpu_manager.go:414] "containerMap: removing container" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" Oct 30 04:32:45.315972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.315969 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31e3220-e28e-432d-9f9a-86ce56e66a78" containerName="istio-proxy" Oct 30 04:32:45.316497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.316062 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" Oct 30 04:32:45.316497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.316073 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" Oct 30 04:32:45.316497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.316082 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d338b3ca-77cc-4e7d-a1f4-202833a479df" containerName="revoke-keys" Oct 30 04:32:45.321116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.321084 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:45.342632 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.341717 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl"] Oct 30 04:32:45.345460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.345439 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:32:45.346311 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.346291 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:32:45.448728 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.448695 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e535d631-gz9gl\" (UID: \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:45.448889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.448847 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e535d631-gz9gl\" (UID: \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:45.448974 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.448957 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jlwn\" (UniqueName: \"kubernetes.io/projected/b4057ead-26ed-4fd9-a18d-01cb76c495e4-kube-api-access-4jlwn\") pod \"maas-api-revoke-sub-e535d631-gz9gl\" (UID: \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:45.550303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.550270 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jlwn\" (UniqueName: \"kubernetes.io/projected/b4057ead-26ed-4fd9-a18d-01cb76c495e4-kube-api-access-4jlwn\") pod \"maas-api-revoke-sub-e535d631-gz9gl\" (UID: \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:45.550498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.550352 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e535d631-gz9gl\" (UID: \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:45.550498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.550480 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e535d631-gz9gl\" (UID: \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:45.551184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.551154 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e535d631-gz9gl\" (UID: \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:45.552745 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.552715 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e535d631-gz9gl\" (UID: \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:45.574073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.574018 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jlwn\" (UniqueName: \"kubernetes.io/projected/b4057ead-26ed-4fd9-a18d-01cb76c495e4-kube-api-access-4jlwn\") pod \"maas-api-revoke-sub-e535d631-gz9gl\" (UID: \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:45.638574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.638551 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:45.952566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:45.952534 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl"] Oct 30 04:32:45.952879 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:32:45.952859 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4057ead_26ed_4fd9_a18d_01cb76c495e4.slice/crio-131b58b465f4a85c09bc2ec5482f8af6e5d8e92dc05d0d8bf6dadadafd468020 WatchSource:0}: Error finding container 131b58b465f4a85c09bc2ec5482f8af6e5d8e92dc05d0d8bf6dadadafd468020: Status 404 returned error can't find the container with id 131b58b465f4a85c09bc2ec5482f8af6e5d8e92dc05d0d8bf6dadadafd468020 Oct 30 04:32:46.014353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:46.014326 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" event={"ID":"b4057ead-26ed-4fd9-a18d-01cb76c495e4","Type":"ContainerStarted","Data":"131b58b465f4a85c09bc2ec5482f8af6e5d8e92dc05d0d8bf6dadadafd468020"} Oct 30 04:32:46.539743 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:46.539659 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:47.022523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:47.022488 4793 generic.go:356] "Generic (PLEG): container finished" podID="b4057ead-26ed-4fd9-a18d-01cb76c495e4" containerID="b7ef2d148bbf2c263257d5d0bdf317a82614b25bc7bbd8ae77375c1fdf5b4d91" exitCode=0 Oct 30 04:32:47.022676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:47.022536 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" event={"ID":"b4057ead-26ed-4fd9-a18d-01cb76c495e4","Type":"ContainerDied","Data":"b7ef2d148bbf2c263257d5d0bdf317a82614b25bc7bbd8ae77375c1fdf5b4d91"} Oct 30 04:32:48.226667 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.226639 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:48.259867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.259844 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:32:48.286926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.286899 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-openshift-ca\") pod \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\" (UID: \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\") " Oct 30 04:32:48.287074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.287056 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4057ead-26ed-4fd9-a18d-01cb76c495e4-kube-api-access-4jlwn\" (UniqueName: \"kubernetes.io/projected/b4057ead-26ed-4fd9-a18d-01cb76c495e4-kube-api-access-4jlwn\") pod \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\" (UID: \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\") " Oct 30 04:32:48.287175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.287159 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-xks-ca\") pod \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\" (UID: \"b4057ead-26ed-4fd9-a18d-01cb76c495e4\") " Oct 30 04:32:48.287551 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.287288 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-openshift-ca" pod "b4057ead-26ed-4fd9-a18d-01cb76c495e4" (UID: "b4057ead-26ed-4fd9-a18d-01cb76c495e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:48.287694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.287676 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.289730 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.289708 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4057ead-26ed-4fd9-a18d-01cb76c495e4-kube-api-access-4jlwn" pod "b4057ead-26ed-4fd9-a18d-01cb76c495e4" (UID: "b4057ead-26ed-4fd9-a18d-01cb76c495e4"). InnerVolumeSpecName "kube-api-access-4jlwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:48.289868 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.289845 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-xks-ca" pod "b4057ead-26ed-4fd9-a18d-01cb76c495e4" (UID: "b4057ead-26ed-4fd9-a18d-01cb76c495e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:48.388219 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388187 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-certs\") pod \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " Oct 30 04:32:48.388393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388248 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-data\") pod \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " Oct 30 04:32:48.388393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388280 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-credential-socket\") pod \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " Oct 30 04:32:48.388393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388328 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istiod-ca-cert\") pod \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " Oct 30 04:32:48.388604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388465 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-kube-api-access-zjkd5\" (UniqueName: \"kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-kube-api-access-zjkd5\") pod \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " Oct 30 04:32:48.388604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388510 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-podinfo\") pod \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " Oct 30 04:32:48.388604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388518 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-certs" pod "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" (UID: "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:48.388604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388529 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-data" pod "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" (UID: "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:48.388604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388565 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-token\" (UniqueName: \"kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-token\") pod \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " Oct 30 04:32:48.388604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388600 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-socket\") pod \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " Oct 30 04:32:48.388975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388650 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-credential-socket" pod "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" (UID: "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:48.388975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388685 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-envoy\") pod \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\" (UID: \"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf\") " Oct 30 04:32:48.388975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388789 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istiod-ca-cert" pod "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" (UID: "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:48.388975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.388882 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-socket" pod "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" (UID: "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:48.389220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.389201 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.389293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.389226 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jlwn\" (UniqueName: \"kubernetes.io/projected/b4057ead-26ed-4fd9-a18d-01cb76c495e4-kube-api-access-4jlwn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.389293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.389240 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.389293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.389255 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4057ead-26ed-4fd9-a18d-01cb76c495e4-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.389293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.389270 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.389293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.389283 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.389519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.389295 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.390702 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.390678 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-token" pod "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" (UID: "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:48.391180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.391158 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-kube-api-access-zjkd5" pod "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" (UID: "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf"). InnerVolumeSpecName "kube-api-access-zjkd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:48.391280 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.391187 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-podinfo" pod "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" (UID: "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:32:48.391280 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.391193 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-envoy" pod "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" (UID: "9f4b2377-59fe-4459-ae4f-7cbbe753a8cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:32:48.490370 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.490291 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjkd5\" (UniqueName: \"kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-kube-api-access-zjkd5\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.490370 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.490322 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.490370 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.490332 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.490370 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.490344 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:48.892299 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:48.892259 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:49.037343 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.037313 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" Oct 30 04:32:49.037668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.037308 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl" event={"ID":"b4057ead-26ed-4fd9-a18d-01cb76c495e4","Type":"ContainerDied","Data":"131b58b465f4a85c09bc2ec5482f8af6e5d8e92dc05d0d8bf6dadadafd468020"} Oct 30 04:32:49.037668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.037547 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="131b58b465f4a85c09bc2ec5482f8af6e5d8e92dc05d0d8bf6dadadafd468020" Oct 30 04:32:49.039682 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.039654 4793 generic.go:356] "Generic (PLEG): container finished" podID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerID="f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336" exitCode=0 Oct 30 04:32:49.039818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.039723 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" Oct 30 04:32:49.039818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.039737 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" event={"ID":"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf","Type":"ContainerDied","Data":"f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336"} Oct 30 04:32:49.039818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.039788 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf" event={"ID":"9f4b2377-59fe-4459-ae4f-7cbbe753a8cf","Type":"ContainerDied","Data":"f0a314bc474e0da8f739265337fcdcb0a5ce2164bd3e951045f4aff3cc61ef6e"} Oct 30 04:32:49.039998 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.039818 4793 scope.go:122] "RemoveContainer" containerID="f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336" Oct 30 04:32:49.050750 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.050733 4793 scope.go:122] "RemoveContainer" containerID="42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04" Oct 30 04:32:49.064243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.064198 4793 scope.go:122] "RemoveContainer" containerID="f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336" Oct 30 04:32:49.064850 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:49.064828 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336\": container with ID starting with f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336 not found: ID does not exist" containerID="f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336" Oct 30 04:32:49.064946 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.064860 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336"} err="failed to get container status \"f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336\": rpc error: code = NotFound desc = could not find container \"f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336\": container with ID starting with f2e001298b1cc12acba9c809f7114e88954a30e7f46b9f5d00aba8ae478a2336 not found: ID does not exist" Oct 30 04:32:49.064946 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.064878 4793 scope.go:122] "RemoveContainer" containerID="42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04" Oct 30 04:32:49.065228 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:32:49.065203 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04\": container with ID starting with 42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04 not found: ID does not exist" containerID="42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04" Oct 30 04:32:49.065299 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.065227 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04"} err="failed to get container status \"42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04\": rpc error: code = NotFound desc = could not find container \"42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04\": container with ID starting with 42364a69880509340a2bc7e51e5efdcc57c5e83a32d6ebf1436ca4c408d99c04 not found: ID does not exist" Oct 30 04:32:49.116030 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.115964 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf"] Oct 30 04:32:49.133802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:49.133763 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3f5508-openshift-default-547b8d6877-qxlzf"] Oct 30 04:32:50.505399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:50.505212 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:50.732266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:50.732231 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:50.847636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:50.847567 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" path="/var/lib/kubelet/pods/9f4b2377-59fe-4459-ae4f-7cbbe753a8cf/volumes" Oct 30 04:32:52.378843 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:52.378802 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:54.083799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:54.083766 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:54.321020 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:54.320982 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:56.137834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.137796 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:56.158944 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.158912 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz"] Oct 30 04:32:56.159802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.159767 4793 cpu_manager.go:414] "containerMap: removing container" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" Oct 30 04:32:56.159802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.159792 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" Oct 30 04:32:56.159984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.159832 4793 cpu_manager.go:414] "containerMap: removing container" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" Oct 30 04:32:56.159984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.159842 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" Oct 30 04:32:56.159984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.159868 4793 cpu_manager.go:414] "containerMap: removing container" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" Oct 30 04:32:56.159984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.159876 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7805bda5-d2a2-4d8c-87a9-306ffa87fa4a" containerName="istio-proxy" Oct 30 04:32:56.159984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.159888 4793 cpu_manager.go:414] "containerMap: removing container" podUID="b4057ead-26ed-4fd9-a18d-01cb76c495e4" containerName="revoke-keys" Oct 30 04:32:56.159984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.159896 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4057ead-26ed-4fd9-a18d-01cb76c495e4" containerName="revoke-keys" Oct 30 04:32:56.160295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.160040 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" Oct 30 04:32:56.160295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.160055 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4057ead-26ed-4fd9-a18d-01cb76c495e4" containerName="revoke-keys" Oct 30 04:32:56.164147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.164124 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:32:56.179074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.179052 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz"] Oct 30 04:32:56.182179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.182156 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:32:56.191500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.191469 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:32:56.293624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.293588 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgkcn\" (UniqueName: \"kubernetes.io/projected/bab1fd5b-73b3-4a6e-8d93-40333314269c-kube-api-access-tgkcn\") pod \"maas-api-revoke-sub-0ceb8735-mb2mz\" (UID: \"bab1fd5b-73b3-4a6e-8d93-40333314269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:32:56.293809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.293630 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ceb8735-mb2mz\" (UID: \"bab1fd5b-73b3-4a6e-8d93-40333314269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:32:56.293876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.293819 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ceb8735-mb2mz\" (UID: \"bab1fd5b-73b3-4a6e-8d93-40333314269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:32:56.395311 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.395219 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgkcn\" (UniqueName: \"kubernetes.io/projected/bab1fd5b-73b3-4a6e-8d93-40333314269c-kube-api-access-tgkcn\") pod \"maas-api-revoke-sub-0ceb8735-mb2mz\" (UID: \"bab1fd5b-73b3-4a6e-8d93-40333314269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:32:56.395311 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.395275 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ceb8735-mb2mz\" (UID: \"bab1fd5b-73b3-4a6e-8d93-40333314269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:32:56.395577 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.395486 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ceb8735-mb2mz\" (UID: \"bab1fd5b-73b3-4a6e-8d93-40333314269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:32:56.396186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.396158 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ceb8735-mb2mz\" (UID: \"bab1fd5b-73b3-4a6e-8d93-40333314269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:32:56.397696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.397668 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ceb8735-mb2mz\" (UID: \"bab1fd5b-73b3-4a6e-8d93-40333314269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:32:56.418631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.418607 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgkcn\" (UniqueName: \"kubernetes.io/projected/bab1fd5b-73b3-4a6e-8d93-40333314269c-kube-api-access-tgkcn\") pod \"maas-api-revoke-sub-0ceb8735-mb2mz\" (UID: \"bab1fd5b-73b3-4a6e-8d93-40333314269c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:32:56.479004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.478977 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:32:56.781234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:56.781202 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz"] Oct 30 04:32:56.785617 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:32:56.785577 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab1fd5b_73b3_4a6e_8d93_40333314269c.slice/crio-ebaa98365f8c46206b6e80027e4d3e23cb52b64eff9d9349bfaf01066953420c WatchSource:0}: Error finding container ebaa98365f8c46206b6e80027e4d3e23cb52b64eff9d9349bfaf01066953420c: Status 404 returned error can't find the container with id ebaa98365f8c46206b6e80027e4d3e23cb52b64eff9d9349bfaf01066953420c Oct 30 04:32:57.099552 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:57.099451 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" event={"ID":"bab1fd5b-73b3-4a6e-8d93-40333314269c","Type":"ContainerStarted","Data":"ebaa98365f8c46206b6e80027e4d3e23cb52b64eff9d9349bfaf01066953420c"} Oct 30 04:32:57.912890 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:57.912855 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:58.109248 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:58.109161 4793 generic.go:356] "Generic (PLEG): container finished" podID="bab1fd5b-73b3-4a6e-8d93-40333314269c" containerID="d11b4337b3b1cefafb6d134bedde5653666abcff26e2cbd015cea17428c9ef3c" exitCode=0 Oct 30 04:32:58.109393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:58.109250 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" event={"ID":"bab1fd5b-73b3-4a6e-8d93-40333314269c","Type":"ContainerDied","Data":"d11b4337b3b1cefafb6d134bedde5653666abcff26e2cbd015cea17428c9ef3c"} Oct 30 04:32:59.286440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.286402 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:32:59.334043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.334015 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-xks-ca\") pod \"bab1fd5b-73b3-4a6e-8d93-40333314269c\" (UID: \"bab1fd5b-73b3-4a6e-8d93-40333314269c\") " Oct 30 04:32:59.334223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.334094 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab1fd5b-73b3-4a6e-8d93-40333314269c-kube-api-access-tgkcn\" (UniqueName: \"kubernetes.io/projected/bab1fd5b-73b3-4a6e-8d93-40333314269c-kube-api-access-tgkcn\") pod \"bab1fd5b-73b3-4a6e-8d93-40333314269c\" (UID: \"bab1fd5b-73b3-4a6e-8d93-40333314269c\") " Oct 30 04:32:59.334223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.334136 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-openshift-ca\") pod \"bab1fd5b-73b3-4a6e-8d93-40333314269c\" (UID: \"bab1fd5b-73b3-4a6e-8d93-40333314269c\") " Oct 30 04:32:59.334578 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.334555 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-openshift-ca" pod "bab1fd5b-73b3-4a6e-8d93-40333314269c" (UID: "bab1fd5b-73b3-4a6e-8d93-40333314269c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:59.336138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.336110 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-xks-ca" pod "bab1fd5b-73b3-4a6e-8d93-40333314269c" (UID: "bab1fd5b-73b3-4a6e-8d93-40333314269c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:32:59.336215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.336183 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab1fd5b-73b3-4a6e-8d93-40333314269c-kube-api-access-tgkcn" pod "bab1fd5b-73b3-4a6e-8d93-40333314269c" (UID: "bab1fd5b-73b3-4a6e-8d93-40333314269c"). InnerVolumeSpecName "kube-api-access-tgkcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:59.367890 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.367821 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg"] Oct 30 04:32:59.368100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.368071 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" podUID="859cce50-f6af-42e6-a3d6-9a4c76f04c7f" containerName="payload-processing" containerID="cri-o://06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665" gracePeriod=30 Oct 30 04:32:59.387816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.385059 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt"] Oct 30 04:32:59.387816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.385347 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" podUID="6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73" containerName="payload-pre-processing" containerID="cri-o://8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942" gracePeriod=30 Oct 30 04:32:59.388326 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.388243 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:59.435726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.435692 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgkcn\" (UniqueName: \"kubernetes.io/projected/bab1fd5b-73b3-4a6e-8d93-40333314269c-kube-api-access-tgkcn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:59.435726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.435719 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:59.435726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.435731 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab1fd5b-73b3-4a6e-8d93-40333314269c-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:59.588023 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.587908 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:32:59.682811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.682784 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" Oct 30 04:32:59.697641 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.697618 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" Oct 30 04:32:59.739226 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.739200 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-kube-api-access-lgxdc\" (UniqueName: \"kubernetes.io/projected/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-kube-api-access-lgxdc\") pod \"859cce50-f6af-42e6-a3d6-9a4c76f04c7f\" (UID: \"859cce50-f6af-42e6-a3d6-9a4c76f04c7f\") " Oct 30 04:32:59.739387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.739243 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-plugins-config-volume\") pod \"859cce50-f6af-42e6-a3d6-9a4c76f04c7f\" (UID: \"859cce50-f6af-42e6-a3d6-9a4c76f04c7f\") " Oct 30 04:32:59.739387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.739288 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-plugins-config-volume\") pod \"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73\" (UID: \"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73\") " Oct 30 04:32:59.739387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.739374 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-kube-api-access-vnxsz\" (UniqueName: \"kubernetes.io/projected/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-kube-api-access-vnxsz\") pod \"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73\" (UID: \"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73\") " Oct 30 04:32:59.739927 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.739897 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-plugins-config-volume" pod "859cce50-f6af-42e6-a3d6-9a4c76f04c7f" (UID: "859cce50-f6af-42e6-a3d6-9a4c76f04c7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:59.740345 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.740316 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-plugins-config-volume" pod "6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73" (UID: "6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:32:59.741638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.741590 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-kube-api-access-lgxdc" pod "859cce50-f6af-42e6-a3d6-9a4c76f04c7f" (UID: "859cce50-f6af-42e6-a3d6-9a4c76f04c7f"). InnerVolumeSpecName "kube-api-access-lgxdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:59.742233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.742200 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-kube-api-access-vnxsz" pod "6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73" (UID: "6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73"). InnerVolumeSpecName "kube-api-access-vnxsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:32:59.844467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.841369 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgxdc\" (UniqueName: \"kubernetes.io/projected/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-kube-api-access-lgxdc\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:59.844467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.841403 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/859cce50-f6af-42e6-a3d6-9a4c76f04c7f-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:59.844467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.841438 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:32:59.844467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:32:59.841453 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnxsz\" (UniqueName: \"kubernetes.io/projected/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73-kube-api-access-vnxsz\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:00.104770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.104721 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:00.125791 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.125756 4793 generic.go:356] "Generic (PLEG): container finished" podID="859cce50-f6af-42e6-a3d6-9a4c76f04c7f" containerID="06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665" exitCode=0 Oct 30 04:33:00.125955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.125870 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" Oct 30 04:33:00.125955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.125897 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" event={"ID":"859cce50-f6af-42e6-a3d6-9a4c76f04c7f","Type":"ContainerDied","Data":"06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665"} Oct 30 04:33:00.126076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.125958 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg" event={"ID":"859cce50-f6af-42e6-a3d6-9a4c76f04c7f","Type":"ContainerDied","Data":"4a9295dd621160364a2c40f64f6d2b195422dd510a95cc54ac52362f2abbe5bf"} Oct 30 04:33:00.126076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.125992 4793 scope.go:122] "RemoveContainer" containerID="06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665" Oct 30 04:33:00.127764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.127721 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" Oct 30 04:33:00.127900 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.127770 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz" event={"ID":"bab1fd5b-73b3-4a6e-8d93-40333314269c","Type":"ContainerDied","Data":"ebaa98365f8c46206b6e80027e4d3e23cb52b64eff9d9349bfaf01066953420c"} Oct 30 04:33:00.127900 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.127851 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebaa98365f8c46206b6e80027e4d3e23cb52b64eff9d9349bfaf01066953420c" Oct 30 04:33:00.132871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.132844 4793 generic.go:356] "Generic (PLEG): container finished" podID="6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73" containerID="8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942" exitCode=0 Oct 30 04:33:00.132971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.132878 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" Oct 30 04:33:00.132971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.132916 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" event={"ID":"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73","Type":"ContainerDied","Data":"8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942"} Oct 30 04:33:00.132971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.132958 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt" event={"ID":"6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73","Type":"ContainerDied","Data":"cbbc92151524b41b2ef554e74dc9e64e1808dd3c58cad0d97b36901d158abd66"} Oct 30 04:33:00.139851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.139825 4793 scope.go:122] "RemoveContainer" containerID="06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665" Oct 30 04:33:00.142880 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:33:00.140402 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665\": container with ID starting with 06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665 not found: ID does not exist" containerID="06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665" Oct 30 04:33:00.142880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.140474 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665"} err="failed to get container status \"06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665\": rpc error: code = NotFound desc = could not find container \"06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665\": container with ID starting with 06efd82aca01ed0d8a3d3b2ca3fbbbb1d2b8e5e79d3b2f5eec7237939bdf1665 not found: ID does not exist" Oct 30 04:33:00.142880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.140500 4793 scope.go:122] "RemoveContainer" containerID="8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942" Oct 30 04:33:00.164535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.164513 4793 scope.go:122] "RemoveContainer" containerID="8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942" Oct 30 04:33:00.164801 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:33:00.164779 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942\": container with ID starting with 8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942 not found: ID does not exist" containerID="8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942" Oct 30 04:33:00.164855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.164810 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942"} err="failed to get container status \"8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942\": rpc error: code = NotFound desc = could not find container \"8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942\": container with ID starting with 8f1852f8e193270bdcb1e27529b41c5ba1f83ddcc68decebbfa75bf0f5941942 not found: ID does not exist" Oct 30 04:33:00.182308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.182286 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg"] Oct 30 04:33:00.186396 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.186373 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6a777c-6454686bbb-bqxvg"] Oct 30 04:33:00.237589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.237560 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt"] Oct 30 04:33:00.241274 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.241249 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6a777c-69c6c8467-q4pvt"] Oct 30 04:33:00.845238 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.845207 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73" path="/var/lib/kubelet/pods/6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73/volumes" Oct 30 04:33:00.845660 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:00.845646 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="859cce50-f6af-42e6-a3d6-9a4c76f04c7f" path="/var/lib/kubelet/pods/859cce50-f6af-42e6-a3d6-9a4c76f04c7f/volumes" Oct 30 04:33:01.783258 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:01.783226 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:03.351954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:03.351914 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:03.599075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:03.599037 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:05.133851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:05.133816 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:05.338161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:05.338129 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:05.639015 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:05.638979 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:05.844736 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:05.844702 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:06.331439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:06.330915 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:07.979595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:07.979557 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:08.193781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.193730 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:08.586689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.586332 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x"] Oct 30 04:33:08.587183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.587078 4793 cpu_manager.go:414] "containerMap: removing container" podUID="6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73" containerName="payload-pre-processing" Oct 30 04:33:08.587183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.587101 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73" containerName="payload-pre-processing" Oct 30 04:33:08.587183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.587135 4793 cpu_manager.go:414] "containerMap: removing container" podUID="859cce50-f6af-42e6-a3d6-9a4c76f04c7f" containerName="payload-processing" Oct 30 04:33:08.587183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.587144 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="859cce50-f6af-42e6-a3d6-9a4c76f04c7f" containerName="payload-processing" Oct 30 04:33:08.587183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.587167 4793 cpu_manager.go:414] "containerMap: removing container" podUID="bab1fd5b-73b3-4a6e-8d93-40333314269c" containerName="revoke-keys" Oct 30 04:33:08.587183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.587174 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab1fd5b-73b3-4a6e-8d93-40333314269c" containerName="revoke-keys" Oct 30 04:33:08.587586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.587319 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="859cce50-f6af-42e6-a3d6-9a4c76f04c7f" containerName="payload-processing" Oct 30 04:33:08.587586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.587339 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab1fd5b-73b3-4a6e-8d93-40333314269c" containerName="revoke-keys" Oct 30 04:33:08.587586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.587354 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4b2377-59fe-4459-ae4f-7cbbe753a8cf" containerName="istio-proxy" Oct 30 04:33:08.587586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.587368 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5cdbeb-cea3-4d84-befb-57c5fe7c6b73" containerName="payload-pre-processing" Oct 30 04:33:08.592728 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.592706 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.612111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.612092 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-86eaa9-openshift-default-dockercfg-5psm5\"" Oct 30 04:33:08.627117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.627088 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x"] Oct 30 04:33:08.651899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.651868 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-socket\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.658838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.658801 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-token\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.658949 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.658869 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-certs\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.658949 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.658925 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-podinfo\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.659071 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.658979 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bglgf\" (UniqueName: \"kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-kube-api-access-bglgf\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.659127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.659088 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-credential-socket\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.659221 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.659200 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-data\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.659284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.659270 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-envoy\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.659365 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.659353 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8095c1-91a7-4082-84d2-e52b612e736b-istiod-ca-cert\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.767656 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-socket\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.767708 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-token\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.767765 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-certs\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.767807 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-podinfo\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.767855 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bglgf\" (UniqueName: \"kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-kube-api-access-bglgf\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.767863 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.767921 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-credential-socket\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.768013 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-data\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.768086 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-envoy\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.768173 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8095c1-91a7-4082-84d2-e52b612e736b-istiod-ca-cert\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.768313 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-certs\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.769099 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-data\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.769448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.769367 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-credential-socket\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.770201 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.769649 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8095c1-91a7-4082-84d2-e52b612e736b-istiod-ca-cert\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.773449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.773350 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-podinfo\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.776804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.776771 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-socket\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.780516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.780489 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-envoy\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.792937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.792885 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-token\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.793067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.793041 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bglgf\" (UniqueName: \"kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-kube-api-access-bglgf\") pod \"e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.907154 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.907060 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:08.970969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:08.970932 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:09.231817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.231788 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x"] Oct 30 04:33:09.232180 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:33:09.231884 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac8095c1_91a7_4082_84d2_e52b612e736b.slice/crio-6548b5c0a864e16d26e3d6569b0788d01cbbca50463da7bffd622165c453e71d WatchSource:0}: Error finding container 6548b5c0a864e16d26e3d6569b0788d01cbbca50463da7bffd622165c453e71d: Status 404 returned error can't find the container with id 6548b5c0a864e16d26e3d6569b0788d01cbbca50463da7bffd622165c453e71d Oct 30 04:33:09.234126 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.234091 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:09.234260 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.234168 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:09.234260 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.234213 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:09.768227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.768162 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88"] Oct 30 04:33:09.781060 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.781027 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:09.791134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.791103 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88"] Oct 30 04:33:09.795319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.795292 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:09.796472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.796450 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:33:09.796582 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.796490 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:33:09.880431 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.880385 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e39b40-h7q88\" (UID: \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:09.880586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.880532 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e39b40-h7q88\" (UID: \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:09.880686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.880670 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8sx\" (UniqueName: \"kubernetes.io/projected/04fe975f-76ed-4576-882d-9f5bdcd32b9a-kube-api-access-8g8sx\") pod \"maas-api-revoke-sub-d4e39b40-h7q88\" (UID: \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:09.982142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.982111 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e39b40-h7q88\" (UID: \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:09.982333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.982234 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8sx\" (UniqueName: \"kubernetes.io/projected/04fe975f-76ed-4576-882d-9f5bdcd32b9a-kube-api-access-8g8sx\") pod \"maas-api-revoke-sub-d4e39b40-h7q88\" (UID: \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:09.982403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.982346 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e39b40-h7q88\" (UID: \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:09.982921 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.982896 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e39b40-h7q88\" (UID: \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:09.984728 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:09.984706 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e39b40-h7q88\" (UID: \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:10.005873 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:10.005848 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8sx\" (UniqueName: \"kubernetes.io/projected/04fe975f-76ed-4576-882d-9f5bdcd32b9a-kube-api-access-8g8sx\") pod \"maas-api-revoke-sub-d4e39b40-h7q88\" (UID: \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:10.103779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:10.103704 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:10.216470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:10.216430 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" event={"ID":"ac8095c1-91a7-4082-84d2-e52b612e736b","Type":"ContainerStarted","Data":"1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63"} Oct 30 04:33:10.216652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:10.216494 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" event={"ID":"ac8095c1-91a7-4082-84d2-e52b612e736b","Type":"ContainerStarted","Data":"6548b5c0a864e16d26e3d6569b0788d01cbbca50463da7bffd622165c453e71d"} Oct 30 04:33:10.262913 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:10.262866 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" podStartSLOduration=2.262852107 podStartE2EDuration="2.262852107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:10.262072574 +0000 UTC m=+2113.547947370" watchObservedRunningTime="2026-10-30 04:33:10.262852107 +0000 UTC m=+2113.548726883" Oct 30 04:33:10.409007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:10.408977 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88"] Oct 30 04:33:10.410720 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:33:10.410688 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fe975f_76ed_4576_882d_9f5bdcd32b9a.slice/crio-affff3b46471ed0a533094dceb5f7ded141e1cfd433b9d5ae8ee424f26be92dd WatchSource:0}: Error finding container affff3b46471ed0a533094dceb5f7ded141e1cfd433b9d5ae8ee424f26be92dd: Status 404 returned error can't find the container with id affff3b46471ed0a533094dceb5f7ded141e1cfd433b9d5ae8ee424f26be92dd Oct 30 04:33:10.907931 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:10.907899 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:10.914356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:10.914329 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:11.226715 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:11.226682 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" event={"ID":"04fe975f-76ed-4576-882d-9f5bdcd32b9a","Type":"ContainerStarted","Data":"218c69f6b1f8b1f6128ee7f51eb592968a2f1505b43217d6a99af0d76d588e6c"} Oct 30 04:33:11.226875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:11.226741 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" event={"ID":"04fe975f-76ed-4576-882d-9f5bdcd32b9a","Type":"ContainerStarted","Data":"affff3b46471ed0a533094dceb5f7ded141e1cfd433b9d5ae8ee424f26be92dd"} Oct 30 04:33:11.227283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:11.227255 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:11.228023 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:33:11.227993 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:33:11.228140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:11.228095 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:11.268787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:11.268730 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" podStartSLOduration=2.268713425 podStartE2EDuration="2.268713425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:11.267095587 +0000 UTC m=+2114.552970364" watchObservedRunningTime="2026-10-30 04:33:11.268713425 +0000 UTC m=+2114.554588201" Oct 30 04:33:11.714885 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:11.714847 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:12.246634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:12.246596 4793 generic.go:356] "Generic (PLEG): container finished" podID="04fe975f-76ed-4576-882d-9f5bdcd32b9a" containerID="218c69f6b1f8b1f6128ee7f51eb592968a2f1505b43217d6a99af0d76d588e6c" exitCode=0 Oct 30 04:33:12.246791 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:12.246673 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" event={"ID":"04fe975f-76ed-4576-882d-9f5bdcd32b9a","Type":"ContainerDied","Data":"218c69f6b1f8b1f6128ee7f51eb592968a2f1505b43217d6a99af0d76d588e6c"} Oct 30 04:33:12.929411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:12.929376 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:13.134808 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:13.134754 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:13.409689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:13.409667 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:13.528684 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:13.528591 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-xks-ca\") pod \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\" (UID: \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\") " Oct 30 04:33:13.528858 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:13.528698 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fe975f-76ed-4576-882d-9f5bdcd32b9a-kube-api-access-8g8sx\" (UniqueName: \"kubernetes.io/projected/04fe975f-76ed-4576-882d-9f5bdcd32b9a-kube-api-access-8g8sx\") pod \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\" (UID: \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\") " Oct 30 04:33:13.528858 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:13.528815 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-openshift-ca\") pod \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\" (UID: \"04fe975f-76ed-4576-882d-9f5bdcd32b9a\") " Oct 30 04:33:13.529779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:13.529721 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-openshift-ca" pod "04fe975f-76ed-4576-882d-9f5bdcd32b9a" (UID: "04fe975f-76ed-4576-882d-9f5bdcd32b9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:33:13.533195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:13.533164 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-xks-ca" pod "04fe975f-76ed-4576-882d-9f5bdcd32b9a" (UID: "04fe975f-76ed-4576-882d-9f5bdcd32b9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:33:13.533399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:13.533376 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fe975f-76ed-4576-882d-9f5bdcd32b9a-kube-api-access-8g8sx" pod "04fe975f-76ed-4576-882d-9f5bdcd32b9a" (UID: "04fe975f-76ed-4576-882d-9f5bdcd32b9a"). InnerVolumeSpecName "kube-api-access-8g8sx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:33:13.604229 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:13.604191 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:13.630678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:13.630646 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:13.630678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:13.630674 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g8sx\" (UniqueName: \"kubernetes.io/projected/04fe975f-76ed-4576-882d-9f5bdcd32b9a-kube-api-access-8g8sx\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:13.630853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:13.630687 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04fe975f-76ed-4576-882d-9f5bdcd32b9a-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:14.262390 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.262353 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" event={"ID":"04fe975f-76ed-4576-882d-9f5bdcd32b9a","Type":"ContainerDied","Data":"affff3b46471ed0a533094dceb5f7ded141e1cfd433b9d5ae8ee424f26be92dd"} Oct 30 04:33:14.262901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.262411 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="affff3b46471ed0a533094dceb5f7ded141e1cfd433b9d5ae8ee424f26be92dd" Oct 30 04:33:14.262901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.262365 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88" Oct 30 04:33:14.590478 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.590370 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:14.605390 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.605359 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt"] Oct 30 04:33:14.605903 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.605889 4793 cpu_manager.go:414] "containerMap: removing container" podUID="04fe975f-76ed-4576-882d-9f5bdcd32b9a" containerName="revoke-keys" Oct 30 04:33:14.605945 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.605904 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fe975f-76ed-4576-882d-9f5bdcd32b9a" containerName="revoke-keys" Oct 30 04:33:14.606023 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.606013 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fe975f-76ed-4576-882d-9f5bdcd32b9a" containerName="revoke-keys" Oct 30 04:33:14.609562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.609546 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" Oct 30 04:33:14.624560 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.624536 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-86eaa9\"" Oct 30 04:33:14.628460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.628436 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt"] Oct 30 04:33:14.668966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.668931 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p"] Oct 30 04:33:14.673173 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.673149 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" Oct 30 04:33:14.689481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.689454 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-86eaa9-dockercfg-z8zms\"" Oct 30 04:33:14.692162 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.692121 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p"] Oct 30 04:33:14.762440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.761266 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53227857-f430-41a9-9a08-3e2daad9f1b7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt\" (UID: \"53227857-f430-41a9-9a08-3e2daad9f1b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" Oct 30 04:33:14.762440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.761409 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cw2v\" (UniqueName: \"kubernetes.io/projected/53227857-f430-41a9-9a08-3e2daad9f1b7-kube-api-access-6cw2v\") pod \"payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt\" (UID: \"53227857-f430-41a9-9a08-3e2daad9f1b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" Oct 30 04:33:14.799139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.799104 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:14.862516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.862409 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53227857-f430-41a9-9a08-3e2daad9f1b7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt\" (UID: \"53227857-f430-41a9-9a08-3e2daad9f1b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" Oct 30 04:33:14.862690 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.862528 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jg9x\" (UniqueName: \"kubernetes.io/projected/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-kube-api-access-2jg9x\") pod \"payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p\" (UID: \"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" Oct 30 04:33:14.862690 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.862642 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cw2v\" (UniqueName: \"kubernetes.io/projected/53227857-f430-41a9-9a08-3e2daad9f1b7-kube-api-access-6cw2v\") pod \"payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt\" (UID: \"53227857-f430-41a9-9a08-3e2daad9f1b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" Oct 30 04:33:14.862859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.862835 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p\" (UID: \"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" Oct 30 04:33:14.863214 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.863193 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53227857-f430-41a9-9a08-3e2daad9f1b7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt\" (UID: \"53227857-f430-41a9-9a08-3e2daad9f1b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" Oct 30 04:33:14.888925 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.888902 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cw2v\" (UniqueName: \"kubernetes.io/projected/53227857-f430-41a9-9a08-3e2daad9f1b7-kube-api-access-6cw2v\") pod \"payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt\" (UID: \"53227857-f430-41a9-9a08-3e2daad9f1b7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" Oct 30 04:33:14.922984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.922959 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" Oct 30 04:33:14.967139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.962518 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk"] Oct 30 04:33:14.967139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.964043 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p\" (UID: \"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" Oct 30 04:33:14.967139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.964117 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jg9x\" (UniqueName: \"kubernetes.io/projected/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-kube-api-access-2jg9x\") pod \"payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p\" (UID: \"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" Oct 30 04:33:14.967139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.965207 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p\" (UID: \"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" Oct 30 04:33:14.970014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:14.969991 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.009946 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.008344 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk"] Oct 30 04:33:15.015687 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.015503 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-36bbac-openshift-default-dockercfg-n6t7k\"" Oct 30 04:33:15.029459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.027325 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jg9x\" (UniqueName: \"kubernetes.io/projected/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-kube-api-access-2jg9x\") pod \"payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p\" (UID: \"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" Oct 30 04:33:15.065067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.065026 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.065205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.065099 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.065205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.065133 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.065205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.065166 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.065316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.065266 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.066455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.065374 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.066455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.065407 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl55d\" (UniqueName: \"kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-kube-api-access-tl55d\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.066455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.065457 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.066455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.065734 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.167497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.167404 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.167645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.167541 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.167645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.167587 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl55d\" (UniqueName: \"kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-kube-api-access-tl55d\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.167645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.167619 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.167767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.167692 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.167767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.167746 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.167871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.167838 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.167924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.167902 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.167973 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.167943 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.168018 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.167979 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.168018 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.168010 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.168100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.168059 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.168200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.168181 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.168510 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.168486 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.170120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.170095 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.170449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.170405 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.191239 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.191211 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.191414 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.191397 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl55d\" (UniqueName: \"kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-kube-api-access-tl55d\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.247115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.247074 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt"] Oct 30 04:33:15.249114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.249002 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:15.251842 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:33:15.251814 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53227857_f430_41a9_9a08_3e2daad9f1b7.slice/crio-5b36cc4a9b6bd2a95dd59b3dccbce6b0c8c308f938dec3b197fbd8d6449b9a02 WatchSource:0}: Error finding container 5b36cc4a9b6bd2a95dd59b3dccbce6b0c8c308f938dec3b197fbd8d6449b9a02: Status 404 returned error can't find the container with id 5b36cc4a9b6bd2a95dd59b3dccbce6b0c8c308f938dec3b197fbd8d6449b9a02 Oct 30 04:33:15.271657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.271630 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" event={"ID":"53227857-f430-41a9-9a08-3e2daad9f1b7","Type":"ContainerStarted","Data":"5b36cc4a9b6bd2a95dd59b3dccbce6b0c8c308f938dec3b197fbd8d6449b9a02"} Oct 30 04:33:15.291884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.291547 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" Oct 30 04:33:15.316205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.316176 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:15.616134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.616088 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p"] Oct 30 04:33:15.617070 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:33:15.617047 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafd8e6b_2d91_40b5_84fc_e4ddc3b029d7.slice/crio-b1b8653f258806789d0cc774e0457cb40a2171d19393b77c762c8d56c5b14168 WatchSource:0}: Error finding container b1b8653f258806789d0cc774e0457cb40a2171d19393b77c762c8d56c5b14168: Status 404 returned error can't find the container with id b1b8653f258806789d0cc774e0457cb40a2171d19393b77c762c8d56c5b14168 Oct 30 04:33:15.656590 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:33:15.656554 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd66ef7_63fb_42ee_8302_5f04a5c90274.slice/crio-6d2048f6ff27443666ccc1f3a917e1ec0fff813237c404944df67894535859ae WatchSource:0}: Error finding container 6d2048f6ff27443666ccc1f3a917e1ec0fff813237c404944df67894535859ae: Status 404 returned error can't find the container with id 6d2048f6ff27443666ccc1f3a917e1ec0fff813237c404944df67894535859ae Oct 30 04:33:15.658713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.658688 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk"] Oct 30 04:33:15.659057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.659025 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:15.659134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.659104 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:15.659197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:15.659149 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:16.288675 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.288637 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" event={"ID":"dcd66ef7-63fb-42ee-8302-5f04a5c90274","Type":"ContainerStarted","Data":"e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70"} Oct 30 04:33:16.289307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.288702 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" event={"ID":"dcd66ef7-63fb-42ee-8302-5f04a5c90274","Type":"ContainerStarted","Data":"6d2048f6ff27443666ccc1f3a917e1ec0fff813237c404944df67894535859ae"} Oct 30 04:33:16.290389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.290358 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" event={"ID":"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7","Type":"ContainerStarted","Data":"50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee"} Oct 30 04:33:16.290542 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.290413 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" Oct 30 04:33:16.290542 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.290455 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" event={"ID":"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7","Type":"ContainerStarted","Data":"b1b8653f258806789d0cc774e0457cb40a2171d19393b77c762c8d56c5b14168"} Oct 30 04:33:16.292152 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.292123 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" event={"ID":"53227857-f430-41a9-9a08-3e2daad9f1b7","Type":"ContainerStarted","Data":"57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e"} Oct 30 04:33:16.292277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.292194 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" Oct 30 04:33:16.316818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.316792 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:16.318734 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.318709 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 04:33:16.318879 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.318760 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" podUID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 04:33:16.338205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.338155 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" podStartSLOduration=2.3381425670000002 podStartE2EDuration="2.338142567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:16.334132609 +0000 UTC m=+2119.620007376" watchObservedRunningTime="2026-10-30 04:33:16.338142567 +0000 UTC m=+2119.624017393" Oct 30 04:33:16.376436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.376365 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" podStartSLOduration=2.376345848 podStartE2EDuration="2.376345848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:16.37581572 +0000 UTC m=+2119.661690518" watchObservedRunningTime="2026-10-30 04:33:16.376345848 +0000 UTC m=+2119.662220638" Oct 30 04:33:16.426546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:16.426472 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" podStartSLOduration=2.426454716 podStartE2EDuration="2.426454716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:16.41766417 +0000 UTC m=+2119.703538946" watchObservedRunningTime="2026-10-30 04:33:16.426454716 +0000 UTC m=+2119.712329492" Oct 30 04:33:17.002210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:17.002175 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:17.317003 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:17.316920 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 04:33:17.317003 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:17.316975 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" podUID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 04:33:18.316947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:18.316906 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 04:33:18.317154 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:18.316955 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" podUID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 04:33:18.716084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:18.716042 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:19.316625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.316587 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 04:33:19.316825 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.316650 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" podUID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 04:33:19.521408 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.521372 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g"] Oct 30 04:33:19.525664 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.525647 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:19.543046 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.543017 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g"] Oct 30 04:33:19.551393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.551373 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:33:19.551576 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.551475 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:33:19.638991 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.638902 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fxls\" (UniqueName: \"kubernetes.io/projected/f1bfac90-d100-42fa-bfa7-60f4ece89fea-kube-api-access-9fxls\") pod \"maas-api-revoke-sub-31797ec2-dvs2g\" (UID: \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:19.638991 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.638969 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31797ec2-dvs2g\" (UID: \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:19.639174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.639076 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31797ec2-dvs2g\" (UID: \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:19.740536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.740493 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fxls\" (UniqueName: \"kubernetes.io/projected/f1bfac90-d100-42fa-bfa7-60f4ece89fea-kube-api-access-9fxls\") pod \"maas-api-revoke-sub-31797ec2-dvs2g\" (UID: \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:19.740732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.740559 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31797ec2-dvs2g\" (UID: \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:19.740732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.740615 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31797ec2-dvs2g\" (UID: \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:19.741131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.741107 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31797ec2-dvs2g\" (UID: \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:19.742835 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.742812 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31797ec2-dvs2g\" (UID: \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:19.768091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.768064 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fxls\" (UniqueName: \"kubernetes.io/projected/f1bfac90-d100-42fa-bfa7-60f4ece89fea-kube-api-access-9fxls\") pod \"maas-api-revoke-sub-31797ec2-dvs2g\" (UID: \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:19.839825 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.839794 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:19.983442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:19.948981 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:20.213181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:20.212998 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g"] Oct 30 04:33:20.220861 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:33:20.220811 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1bfac90_d100_42fa_bfa7_60f4ece89fea.slice/crio-8e248cdf5d53c1c0e6eff46ca63867ef1149c5e53238134358cc2b168e3effad WatchSource:0}: Error finding container 8e248cdf5d53c1c0e6eff46ca63867ef1149c5e53238134358cc2b168e3effad: Status 404 returned error can't find the container with id 8e248cdf5d53c1c0e6eff46ca63867ef1149c5e53238134358cc2b168e3effad Oct 30 04:33:20.317719 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:20.317674 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 04:33:20.317903 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:20.317738 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" podUID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 04:33:20.327538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:20.327451 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" event={"ID":"f1bfac90-d100-42fa-bfa7-60f4ece89fea","Type":"ContainerStarted","Data":"8e248cdf5d53c1c0e6eff46ca63867ef1149c5e53238134358cc2b168e3effad"} Oct 30 04:33:20.652465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:20.652403 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:21.317091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:21.317046 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 04:33:21.317272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:21.317105 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" podUID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 04:33:21.336671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:21.336639 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" event={"ID":"f1bfac90-d100-42fa-bfa7-60f4ece89fea","Type":"ContainerStarted","Data":"89c4db37ad53f6890a1f2c45ffbc096b68345ea385be23f3340fe14ebb2e2c72"} Oct 30 04:33:21.398498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:21.395447 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" podStartSLOduration=2.3954111129999998 podStartE2EDuration="2.395411113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:21.380106555 +0000 UTC m=+2124.665981331" watchObservedRunningTime="2026-10-30 04:33:21.395411113 +0000 UTC m=+2124.681285891" Oct 30 04:33:22.316731 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:22.316693 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 04:33:22.317149 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:22.316742 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" podUID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 04:33:22.348994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:22.348963 4793 generic.go:356] "Generic (PLEG): container finished" podID="f1bfac90-d100-42fa-bfa7-60f4ece89fea" containerID="89c4db37ad53f6890a1f2c45ffbc096b68345ea385be23f3340fe14ebb2e2c72" exitCode=0 Oct 30 04:33:22.349169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:22.349050 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" event={"ID":"f1bfac90-d100-42fa-bfa7-60f4ece89fea","Type":"ContainerDied","Data":"89c4db37ad53f6890a1f2c45ffbc096b68345ea385be23f3340fe14ebb2e2c72"} Oct 30 04:33:22.625598 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:22.625521 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:22.853204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:22.853156 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:23.662197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:23.661818 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:23.746038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:23.746001 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-xks-ca\") pod \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\" (UID: \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\") " Oct 30 04:33:23.746214 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:23.746089 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-openshift-ca\") pod \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\" (UID: \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\") " Oct 30 04:33:23.746310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:23.746276 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1bfac90-d100-42fa-bfa7-60f4ece89fea-kube-api-access-9fxls\" (UniqueName: \"kubernetes.io/projected/f1bfac90-d100-42fa-bfa7-60f4ece89fea-kube-api-access-9fxls\") pod \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\" (UID: \"f1bfac90-d100-42fa-bfa7-60f4ece89fea\") " Oct 30 04:33:23.746533 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:23.746507 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-openshift-ca" pod "f1bfac90-d100-42fa-bfa7-60f4ece89fea" (UID: "f1bfac90-d100-42fa-bfa7-60f4ece89fea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:33:23.747230 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:23.747203 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:23.748585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:23.748561 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-xks-ca" pod "f1bfac90-d100-42fa-bfa7-60f4ece89fea" (UID: "f1bfac90-d100-42fa-bfa7-60f4ece89fea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:33:23.748665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:23.748585 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1bfac90-d100-42fa-bfa7-60f4ece89fea-kube-api-access-9fxls" pod "f1bfac90-d100-42fa-bfa7-60f4ece89fea" (UID: "f1bfac90-d100-42fa-bfa7-60f4ece89fea"). InnerVolumeSpecName "kube-api-access-9fxls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:33:23.848777 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:23.848751 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fxls\" (UniqueName: \"kubernetes.io/projected/f1bfac90-d100-42fa-bfa7-60f4ece89fea-kube-api-access-9fxls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:23.848906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:23.848780 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1bfac90-d100-42fa-bfa7-60f4ece89fea-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:24.019955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:24.017836 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:24.020148 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:24.019970 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:24.020720 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:24.020682 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:33:24.364182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:24.364094 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" event={"ID":"f1bfac90-d100-42fa-bfa7-60f4ece89fea","Type":"ContainerDied","Data":"8e248cdf5d53c1c0e6eff46ca63867ef1149c5e53238134358cc2b168e3effad"} Oct 30 04:33:24.364182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:24.364153 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e248cdf5d53c1c0e6eff46ca63867ef1149c5e53238134358cc2b168e3effad" Oct 30 04:33:24.364182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:24.364103 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g" Oct 30 04:33:24.998489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:24.998455 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:25.687026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:25.686995 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:26.686854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:26.686815 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:27.303340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:27.303309 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" Oct 30 04:33:27.303566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:27.303384 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" Oct 30 04:33:28.282591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:28.282528 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:28.576185 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:28.576096 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:28.839609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:28.839516 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:30.277246 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:30.277211 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:30.560058 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:30.560027 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948"] Oct 30 04:33:30.723476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:30.723444 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:32.005538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:32.005501 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:33.820872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:33.820822 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:35.141005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:35.140966 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:35.491989 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:35.491950 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:37.176576 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:37.176536 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:38.461254 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.461222 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w"] Oct 30 04:33:38.461988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.461967 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f1bfac90-d100-42fa-bfa7-60f4ece89fea" containerName="revoke-keys" Oct 30 04:33:38.462113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.461990 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1bfac90-d100-42fa-bfa7-60f4ece89fea" containerName="revoke-keys" Oct 30 04:33:38.462157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.462137 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1bfac90-d100-42fa-bfa7-60f4ece89fea" containerName="revoke-keys" Oct 30 04:33:38.466138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.466118 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.506797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.506765 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w"] Oct 30 04:33:38.560445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.560403 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.560637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.560452 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.560637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.560559 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.560637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.560596 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.560818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.560692 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.560818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.560721 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.560818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.560759 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.560978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.560870 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lph4\" (UniqueName: \"kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-kube-api-access-2lph4\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.560978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.560969 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.661943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.661907 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.661943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.661944 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.662167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.661995 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.662167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.662096 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.662167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.662152 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.662340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.662246 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lph4\" (UniqueName: \"kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-kube-api-access-2lph4\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.662395 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.662370 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.662470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.662396 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-certs\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.662589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.662477 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.662589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.662508 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.663036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.662729 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.663036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.662786 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istiod-ca-cert\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.663036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.662909 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-credential-socket\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.663036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.662949 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-data\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.664714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.664693 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-envoy\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.664822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.664763 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-podinfo\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.688049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.688018 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-token\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.688222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.688205 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lph4\" (UniqueName: \"kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-kube-api-access-2lph4\") pod \"e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.781295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.781226 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:38.905339 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:38.905137 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:39.116952 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:33:39.116924 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8b1b7d_8c0b_41ef_964f_42c406f53364.slice/crio-3ec1de42481d3b9058e6abae6c46606c736d92cb560ad3aa40058f1ccf08953e WatchSource:0}: Error finding container 3ec1de42481d3b9058e6abae6c46606c736d92cb560ad3aa40058f1ccf08953e: Status 404 returned error can't find the container with id 3ec1de42481d3b9058e6abae6c46606c736d92cb560ad3aa40058f1ccf08953e Oct 30 04:33:39.117283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:39.117261 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w"] Oct 30 04:33:39.119668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:39.119633 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:39.119792 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:39.119710 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:39.119792 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:39.119756 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:39.509334 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:39.509287 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" event={"ID":"2e8b1b7d-8c0b-41ef-964f-42c406f53364","Type":"ContainerStarted","Data":"19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1"} Oct 30 04:33:39.509775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:39.509350 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" event={"ID":"2e8b1b7d-8c0b-41ef-964f-42c406f53364","Type":"ContainerStarted","Data":"3ec1de42481d3b9058e6abae6c46606c736d92cb560ad3aa40058f1ccf08953e"} Oct 30 04:33:39.559097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:39.559036 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" podStartSLOduration=1.55901897 podStartE2EDuration="1.55901897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:39.555155808 +0000 UTC m=+2142.841030607" watchObservedRunningTime="2026-10-30 04:33:39.55901897 +0000 UTC m=+2142.844893746" Oct 30 04:33:39.781799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:39.781722 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:40.616102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:40.616048 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:40.787986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:40.787951 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 04:33:40.788279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:40.788247 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" podUID="2e8b1b7d-8c0b-41ef-964f-42c406f53364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 04:33:41.220209 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:33:41.220176 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:33:41.784442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:41.782720 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 04:33:41.784442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:41.782791 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" podUID="2e8b1b7d-8c0b-41ef-964f-42c406f53364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 04:33:42.788443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:42.785567 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 04:33:42.788443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:42.785625 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" podUID="2e8b1b7d-8c0b-41ef-964f-42c406f53364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 04:33:42.806459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:42.804377 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:43.784784 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:43.784736 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 04:33:43.784993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:43.784800 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" podUID="2e8b1b7d-8c0b-41ef-964f-42c406f53364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 04:33:44.561405 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:44.561370 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:44.785310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:44.784805 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 04:33:44.785310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:44.784922 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" podUID="2e8b1b7d-8c0b-41ef-964f-42c406f53364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 04:33:45.047720 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.047655 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd"] Oct 30 04:33:45.052948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.052926 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.114569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.114525 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd"] Oct 30 04:33:45.139284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.139246 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.163604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.163563 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.167439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.163900 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.177607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.177574 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.194442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.189496 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.202444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.194825 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.232442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.231521 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbjjx\" (UniqueName: \"kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-kube-api-access-fbjjx\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.232442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.231575 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.232442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.231626 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.362498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.357443 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.362498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.357542 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbjjx\" (UniqueName: \"kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-kube-api-access-fbjjx\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.362498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.357578 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.362498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.357625 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.362498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.357657 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.362498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.357737 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.362498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.357834 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.362498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.357933 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.362498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.357962 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.371453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.364263 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-data\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.376589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.372995 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.376589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.373566 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-credential-socket\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.376589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.374383 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istiod-ca-cert\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.376589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.374635 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-certs\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.549761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.549669 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbjjx\" (UniqueName: \"kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-kube-api-access-fbjjx\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.552768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.552734 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-envoy\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.557169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.557139 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-podinfo\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.578440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.572287 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-token\") pod \"e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.585930 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.585862 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" podUID="6cc03e55-64ec-44bb-934f-6555acc3c342" containerName="main" containerID="cri-o://8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1" gracePeriod=45 Oct 30 04:33:45.671188 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:33:45.671151 4793 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cc03e55_64ec_44bb_934f_6555acc3c342.slice/crio-8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1.scope\": RecentStats: unable to find data in memory cache]" Oct 30 04:33:45.710225 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.710196 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:45.753053 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.752726 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" podUID="6cc03e55-64ec-44bb-934f-6555acc3c342" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/ready\": dial tcp 10.133.0.153:8000: connect: connection refused" Oct 30 04:33:45.792875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.792833 4793 patch_prober.go:29] interesting pod/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 04:33:45.793072 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:45.792907 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" podUID="2e8b1b7d-8c0b-41ef-964f-42c406f53364" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 04:33:46.005763 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.005732 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:33:46.101374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.101338 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd"] Oct 30 04:33:46.108877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.108843 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:46.108988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.108921 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:46.108988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.108963 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:46.118644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.118610 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc03e55-64ec-44bb-934f-6555acc3c342-kube-api-access-6rmq7\" (UniqueName: \"kubernetes.io/projected/6cc03e55-64ec-44bb-934f-6555acc3c342-kube-api-access-6rmq7\") pod \"6cc03e55-64ec-44bb-934f-6555acc3c342\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " Oct 30 04:33:46.119206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.119184 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-dshm\") pod \"6cc03e55-64ec-44bb-934f-6555acc3c342\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " Oct 30 04:33:46.119701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.119683 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cc03e55-64ec-44bb-934f-6555acc3c342-tls-certs\" (UniqueName: \"kubernetes.io/secret/6cc03e55-64ec-44bb-934f-6555acc3c342-tls-certs\") pod \"6cc03e55-64ec-44bb-934f-6555acc3c342\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " Oct 30 04:33:46.120149 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.120133 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-tmp-dir\") pod \"6cc03e55-64ec-44bb-934f-6555acc3c342\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " Oct 30 04:33:46.120607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.120500 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-tmp-dir" pod "6cc03e55-64ec-44bb-934f-6555acc3c342" (UID: "6cc03e55-64ec-44bb-934f-6555acc3c342"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:33:46.120759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.120741 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-model-cache\") pod \"6cc03e55-64ec-44bb-934f-6555acc3c342\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " Oct 30 04:33:46.121122 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.121019 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-model-cache" pod "6cc03e55-64ec-44bb-934f-6555acc3c342" (UID: "6cc03e55-64ec-44bb-934f-6555acc3c342"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:33:46.121262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.121245 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-home\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-home\") pod \"6cc03e55-64ec-44bb-934f-6555acc3c342\" (UID: \"6cc03e55-64ec-44bb-934f-6555acc3c342\") " Oct 30 04:33:46.121635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.121527 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-home" pod "6cc03e55-64ec-44bb-934f-6555acc3c342" (UID: "6cc03e55-64ec-44bb-934f-6555acc3c342"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:33:46.122165 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.122143 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:46.122282 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.122269 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:46.122378 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.122367 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:46.133469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.129784 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc03e55-64ec-44bb-934f-6555acc3c342-tls-certs" pod "6cc03e55-64ec-44bb-934f-6555acc3c342" (UID: "6cc03e55-64ec-44bb-934f-6555acc3c342"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:33:46.134617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.134186 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc03e55-64ec-44bb-934f-6555acc3c342-kube-api-access-6rmq7" pod "6cc03e55-64ec-44bb-934f-6555acc3c342" (UID: "6cc03e55-64ec-44bb-934f-6555acc3c342"). InnerVolumeSpecName "kube-api-access-6rmq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:33:46.137311 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.137278 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-dshm" pod "6cc03e55-64ec-44bb-934f-6555acc3c342" (UID: "6cc03e55-64ec-44bb-934f-6555acc3c342"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:33:46.223642 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.223603 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rmq7\" (UniqueName: \"kubernetes.io/projected/6cc03e55-64ec-44bb-934f-6555acc3c342-kube-api-access-6rmq7\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:46.223642 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.223634 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6cc03e55-64ec-44bb-934f-6555acc3c342-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:46.223870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.223647 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cc03e55-64ec-44bb-934f-6555acc3c342-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:46.396499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.395783 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:46.630061 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.629709 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" event={"ID":"f2c0eabb-3612-403a-a5ae-24fbae4c3930","Type":"ContainerStarted","Data":"d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1"} Oct 30 04:33:46.630061 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.629815 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" event={"ID":"f2c0eabb-3612-403a-a5ae-24fbae4c3930","Type":"ContainerStarted","Data":"10aa0afd6d832956bc281418fabac483fc78d721ab59252d6c82d9d60e2e59f2"} Oct 30 04:33:46.633749 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.633682 4793 generic.go:356] "Generic (PLEG): container finished" podID="6cc03e55-64ec-44bb-934f-6555acc3c342" containerID="8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1" exitCode=0 Oct 30 04:33:46.633956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.633924 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" event={"ID":"6cc03e55-64ec-44bb-934f-6555acc3c342","Type":"ContainerDied","Data":"8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1"} Oct 30 04:33:46.634029 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.633990 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" event={"ID":"6cc03e55-64ec-44bb-934f-6555acc3c342","Type":"ContainerDied","Data":"cd6be9e72189bc45d6706b4d9f25556b59d031b6f3c3b8533b82c4aa30468142"} Oct 30 04:33:46.634092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.634035 4793 scope.go:122] "RemoveContainer" containerID="8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1" Oct 30 04:33:46.634401 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.634380 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948" Oct 30 04:33:46.645931 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.645900 4793 scope.go:122] "RemoveContainer" containerID="8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1" Oct 30 04:33:46.646176 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:33:46.646150 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1\": container with ID starting with 8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1 not found: ID does not exist" containerID="8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1" Oct 30 04:33:46.646266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.646180 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1"} err="failed to get container status \"8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1\": rpc error: code = NotFound desc = could not find container \"8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1\": container with ID starting with 8e6d9aecfdab8b1753a4fc80343d94fab1ba927f28595144b598bcf66efc67e1 not found: ID does not exist" Oct 30 04:33:46.678442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.677561 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" podStartSLOduration=1.677543673 podStartE2EDuration="1.677543673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:46.676993778 +0000 UTC m=+2149.962868554" watchObservedRunningTime="2026-10-30 04:33:46.677543673 +0000 UTC m=+2149.963418449" Oct 30 04:33:46.711592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.711490 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:46.729540 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.729511 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948"] Oct 30 04:33:46.736206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.736178 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-153a44/praxis-route-153a44-kserve-68fc94df89-zd948"] Oct 30 04:33:46.881094 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:46.881059 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc03e55-64ec-44bb-934f-6555acc3c342" path="/var/lib/kubelet/pods/6cc03e55-64ec-44bb-934f-6555acc3c342/volumes" Oct 30 04:33:47.151340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:47.151296 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:47.152576 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:47.152453 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:47.153308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:47.153287 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:33:47.703141 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:47.701006 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c"] Oct 30 04:33:47.703141 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:47.701312 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" podUID="234dc359-6934-4120-b625-f6249ea314b6" containerName="maas-api" containerID="cri-o://2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab" gracePeriod=30 Oct 30 04:33:47.721463 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:47.712206 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 04:33:47.721463 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:47.712259 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" podUID="f2c0eabb-3612-403a-a5ae-24fbae4c3930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 04:33:47.934442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:47.915525 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:47.952592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:47.952497 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb"] Oct 30 04:33:47.952767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:47.952740 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" podUID="aa8a7afc-f51a-49ce-8837-0d5f39b03600" containerName="payload-processing" containerID="cri-o://8605fe7a53f0327cd743014d10790c3cbd331b4b5eb583784f814f71725a0b6f" gracePeriod=30 Oct 30 04:33:47.985442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:47.984990 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt"] Oct 30 04:33:47.985442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:47.985220 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" podUID="0f61929c-96a1-4073-8f8c-acb25ad6d040" containerName="payload-pre-processing" containerID="cri-o://ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51" gracePeriod=30 Oct 30 04:33:48.120442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.119937 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:48.525488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.515793 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:33:48.572657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.572619 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:48.616105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.616067 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/234dc359-6934-4120-b625-f6249ea314b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/234dc359-6934-4120-b625-f6249ea314b6-openshift-service-ca\") pod \"234dc359-6934-4120-b625-f6249ea314b6\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " Oct 30 04:33:48.616564 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.616539 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-metrics-certs\") pod \"234dc359-6934-4120-b625-f6249ea314b6\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " Oct 30 04:33:48.616823 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.616804 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-maas-api-tls\") pod \"234dc359-6934-4120-b625-f6249ea314b6\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " Oct 30 04:33:48.617762 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.617731 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234dc359-6934-4120-b625-f6249ea314b6-openshift-service-ca" pod "234dc359-6934-4120-b625-f6249ea314b6" (UID: "234dc359-6934-4120-b625-f6249ea314b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:33:48.618226 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.618201 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/234dc359-6934-4120-b625-f6249ea314b6-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:48.620863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.620833 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-maas-api-tls" pod "234dc359-6934-4120-b625-f6249ea314b6" (UID: "234dc359-6934-4120-b625-f6249ea314b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:33:48.621884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.621859 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-metrics-certs" pod "234dc359-6934-4120-b625-f6249ea314b6" (UID: "234dc359-6934-4120-b625-f6249ea314b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:33:48.730447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.723973 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 04:33:48.730447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.724034 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" podUID="f2c0eabb-3612-403a-a5ae-24fbae4c3930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 04:33:48.730447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.724855 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/234dc359-6934-4120-b625-f6249ea314b6-kube-api-access-h28mj\" (UniqueName: \"kubernetes.io/projected/234dc359-6934-4120-b625-f6249ea314b6-kube-api-access-h28mj\") pod \"234dc359-6934-4120-b625-f6249ea314b6\" (UID: \"234dc359-6934-4120-b625-f6249ea314b6\") " Oct 30 04:33:48.730447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.725432 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:48.730447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.725452 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/234dc359-6934-4120-b625-f6249ea314b6-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:48.732515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.732132 4793 generic.go:356] "Generic (PLEG): container finished" podID="aa8a7afc-f51a-49ce-8837-0d5f39b03600" containerID="8605fe7a53f0327cd743014d10790c3cbd331b4b5eb583784f814f71725a0b6f" exitCode=0 Oct 30 04:33:48.732515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.732163 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234dc359-6934-4120-b625-f6249ea314b6-kube-api-access-h28mj" pod "234dc359-6934-4120-b625-f6249ea314b6" (UID: "234dc359-6934-4120-b625-f6249ea314b6"). InnerVolumeSpecName "kube-api-access-h28mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:33:48.732515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.732288 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" event={"ID":"aa8a7afc-f51a-49ce-8837-0d5f39b03600","Type":"ContainerDied","Data":"8605fe7a53f0327cd743014d10790c3cbd331b4b5eb583784f814f71725a0b6f"} Oct 30 04:33:48.738126 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.738099 4793 generic.go:356] "Generic (PLEG): container finished" podID="234dc359-6934-4120-b625-f6249ea314b6" containerID="2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab" exitCode=0 Oct 30 04:33:48.738253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.738160 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" Oct 30 04:33:48.738253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.738181 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" event={"ID":"234dc359-6934-4120-b625-f6249ea314b6","Type":"ContainerDied","Data":"2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab"} Oct 30 04:33:48.738253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.738241 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c" event={"ID":"234dc359-6934-4120-b625-f6249ea314b6","Type":"ContainerDied","Data":"7078375a9d94729865bb1f5f7880ca02a1f1911f741fa4292c26821e3452204b"} Oct 30 04:33:48.738468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.738270 4793 scope.go:122] "RemoveContainer" containerID="2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab" Oct 30 04:33:48.743338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.743320 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" Oct 30 04:33:48.749845 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.749830 4793 scope.go:122] "RemoveContainer" containerID="2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab" Oct 30 04:33:48.750119 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:33:48.750088 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab\": container with ID starting with 2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab not found: ID does not exist" containerID="2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab" Oct 30 04:33:48.750166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.750144 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab"} err="failed to get container status \"2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab\": rpc error: code = NotFound desc = could not find container \"2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab\": container with ID starting with 2ef3e6507dc43cafeee45686b9109dbf4857268b022ec0cc95086f9d6579bcab not found: ID does not exist" Oct 30 04:33:48.812843 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.812811 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:48.814765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.814736 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c"] Oct 30 04:33:48.821174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.821153 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7a36d-5995fbfff-cw74c"] Oct 30 04:33:48.827768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.827747 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h28mj\" (UniqueName: \"kubernetes.io/projected/234dc359-6934-4120-b625-f6249ea314b6-kube-api-access-h28mj\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:48.847968 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.847944 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="234dc359-6934-4120-b625-f6249ea314b6" path="/var/lib/kubelet/pods/234dc359-6934-4120-b625-f6249ea314b6/volumes" Oct 30 04:33:48.928434 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.928384 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8a7afc-f51a-49ce-8837-0d5f39b03600-kube-api-access-www45\" (UniqueName: \"kubernetes.io/projected/aa8a7afc-f51a-49ce-8837-0d5f39b03600-kube-api-access-www45\") pod \"aa8a7afc-f51a-49ce-8837-0d5f39b03600\" (UID: \"aa8a7afc-f51a-49ce-8837-0d5f39b03600\") " Oct 30 04:33:48.928638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.928480 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8a7afc-f51a-49ce-8837-0d5f39b03600-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8a7afc-f51a-49ce-8837-0d5f39b03600-plugins-config-volume\") pod \"aa8a7afc-f51a-49ce-8837-0d5f39b03600\" (UID: \"aa8a7afc-f51a-49ce-8837-0d5f39b03600\") " Oct 30 04:33:48.929342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.929308 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8a7afc-f51a-49ce-8837-0d5f39b03600-plugins-config-volume" pod "aa8a7afc-f51a-49ce-8837-0d5f39b03600" (UID: "aa8a7afc-f51a-49ce-8837-0d5f39b03600"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:33:48.931314 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:48.931279 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8a7afc-f51a-49ce-8837-0d5f39b03600-kube-api-access-www45" pod "aa8a7afc-f51a-49ce-8837-0d5f39b03600" (UID: "aa8a7afc-f51a-49ce-8837-0d5f39b03600"). InnerVolumeSpecName "kube-api-access-www45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:33:49.030329 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:49.030254 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-www45\" (UniqueName: \"kubernetes.io/projected/aa8a7afc-f51a-49ce-8837-0d5f39b03600-kube-api-access-www45\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:49.030329 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:49.030281 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8a7afc-f51a-49ce-8837-0d5f39b03600-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:49.542545 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:49.542503 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" podUID="aa8a7afc-f51a-49ce-8837-0d5f39b03600" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.134:9004: i/o timeout" Oct 30 04:33:49.713360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:49.712820 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 04:33:49.713360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:49.712896 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" podUID="f2c0eabb-3612-403a-a5ae-24fbae4c3930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 04:33:49.747289 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:49.747253 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" event={"ID":"aa8a7afc-f51a-49ce-8837-0d5f39b03600","Type":"ContainerDied","Data":"491c29fb53a98d884fa8b710266cf7669f01c0e42bdf01fd98dffeee2b43a5e5"} Oct 30 04:33:49.747777 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:49.747356 4793 scope.go:122] "RemoveContainer" containerID="8605fe7a53f0327cd743014d10790c3cbd331b4b5eb583784f814f71725a0b6f" Oct 30 04:33:49.747777 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:49.747269 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb" Oct 30 04:33:49.878502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:49.878463 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb"] Oct 30 04:33:49.889706 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:49.889677 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7a36d-5c4df657fb-p5srb"] Oct 30 04:33:50.113190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.113096 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:50.333410 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.333376 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:50.520892 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.520860 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt"] Oct 30 04:33:50.522042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.522013 4793 cpu_manager.go:414] "containerMap: removing container" podUID="6cc03e55-64ec-44bb-934f-6555acc3c342" containerName="main" Oct 30 04:33:50.522224 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.522209 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc03e55-64ec-44bb-934f-6555acc3c342" containerName="main" Oct 30 04:33:50.522360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.522347 4793 cpu_manager.go:414] "containerMap: removing container" podUID="234dc359-6934-4120-b625-f6249ea314b6" containerName="maas-api" Oct 30 04:33:50.522507 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.522478 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="234dc359-6934-4120-b625-f6249ea314b6" containerName="maas-api" Oct 30 04:33:50.522673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.522658 4793 cpu_manager.go:414] "containerMap: removing container" podUID="aa8a7afc-f51a-49ce-8837-0d5f39b03600" containerName="payload-processing" Oct 30 04:33:50.522782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.522769 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8a7afc-f51a-49ce-8837-0d5f39b03600" containerName="payload-processing" Oct 30 04:33:50.523021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.523005 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8a7afc-f51a-49ce-8837-0d5f39b03600" containerName="payload-processing" Oct 30 04:33:50.523139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.523127 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc03e55-64ec-44bb-934f-6555acc3c342" containerName="main" Oct 30 04:33:50.523227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.523216 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="234dc359-6934-4120-b625-f6249ea314b6" containerName="maas-api" Oct 30 04:33:50.529553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.529532 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.553127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.550800 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v54l2\" (UniqueName: \"kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-kube-api-access-v54l2\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.553127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.550858 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-token\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.553127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.550931 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istiod-ca-cert\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.553127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.550971 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-certs\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.553127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.551000 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-podinfo\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.553127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.551053 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-credential-socket\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.553127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.551119 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-socket\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.553127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.551151 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-envoy\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.553127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.551197 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-data\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.599456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.572607 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-2696d8-openshift-default-dockercfg-2cr54\"" Oct 30 04:33:50.609497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.609466 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt"] Oct 30 04:33:50.679446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.666973 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-socket\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.679446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.667038 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-envoy\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.679446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.667093 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-data\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.679446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.667269 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v54l2\" (UniqueName: \"kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-kube-api-access-v54l2\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.679446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.667305 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-token\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.679446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.667389 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istiod-ca-cert\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.679446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.667450 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-certs\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.679446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.667483 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-podinfo\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.679446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.667535 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-credential-socket\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.703440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.693541 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-credential-socket\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.703672 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.697218 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-socket\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.710447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.704212 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-data\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.718451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.710926 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 04:33:50.718652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.718613 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-envoy\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.718875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.715704 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-certs\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.718982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.718646 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" podUID="f2c0eabb-3612-403a-a5ae-24fbae4c3930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 04:33:50.719117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.715472 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istiod-ca-cert\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.720217 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.720081 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:50.722809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.722783 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v54l2\" (UniqueName: \"kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-kube-api-access-v54l2\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.726074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.726041 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-podinfo\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.822812 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.822736 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-token\") pod \"e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.889538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.888261 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8a7afc-f51a-49ce-8837-0d5f39b03600" path="/var/lib/kubelet/pods/aa8a7afc-f51a-49ce-8837-0d5f39b03600/volumes" Oct 30 04:33:50.916088 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.916054 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:50.921169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:50.921127 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:51.282644 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:33:51.282615 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf9878b_08a0_4f63_8e02_f6b8c25ef88e.slice/crio-5b0d7a7a7f29177c72dbf6c6819304884502e4c62b236d493a61ed0f5fd7a6b7 WatchSource:0}: Error finding container 5b0d7a7a7f29177c72dbf6c6819304884502e4c62b236d493a61ed0f5fd7a6b7: Status 404 returned error can't find the container with id 5b0d7a7a7f29177c72dbf6c6819304884502e4c62b236d493a61ed0f5fd7a6b7 Oct 30 04:33:51.283380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:51.283311 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:51.283897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:51.283873 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt"] Oct 30 04:33:51.285313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:51.285280 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:51.285397 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:51.285348 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:51.285397 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:51.285381 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:51.493466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:51.491033 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:51.710879 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:51.710841 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 04:33:51.711027 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:51.710908 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" podUID="f2c0eabb-3612-403a-a5ae-24fbae4c3930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 04:33:51.845760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:51.845725 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" event={"ID":"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e","Type":"ContainerStarted","Data":"323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e"} Oct 30 04:33:51.846138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:51.845780 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" event={"ID":"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e","Type":"ContainerStarted","Data":"5b0d7a7a7f29177c72dbf6c6819304884502e4c62b236d493a61ed0f5fd7a6b7"} Oct 30 04:33:51.903557 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:51.903491 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" podStartSLOduration=1.9034771579999998 podStartE2EDuration="1.903477158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:51.899745534 +0000 UTC m=+2155.185620355" watchObservedRunningTime="2026-10-30 04:33:51.903477158 +0000 UTC m=+2155.189351932" Oct 30 04:33:51.916158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:51.916133 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:52.096489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:52.096363 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:52.407864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:52.407080 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:52.426052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:52.425933 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm"] Oct 30 04:33:52.426231 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:52.426186 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" containerID="cri-o://068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2" gracePeriod=30 Oct 30 04:33:52.607026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:52.606850 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:52.710710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:52.710674 4793 patch_prober.go:29] interesting pod/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 30 04:33:52.710891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:52.710748 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" podUID="f2c0eabb-3612-403a-a5ae-24fbae4c3930" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 30 04:33:52.923229 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:52.923193 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:53.715119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:53.715090 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:53.715393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:53.715375 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:53.716074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:53.716052 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:33:53.870500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:53.867737 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:53.870500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:53.869091 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:33:54.082541 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.082503 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:54.297663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.297608 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:54.696164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.696125 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:54.741465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.741442 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" Oct 30 04:33:54.788516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.788485 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f61929c-96a1-4073-8f8c-acb25ad6d040-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f61929c-96a1-4073-8f8c-acb25ad6d040-plugins-config-volume\") pod \"0f61929c-96a1-4073-8f8c-acb25ad6d040\" (UID: \"0f61929c-96a1-4073-8f8c-acb25ad6d040\") " Oct 30 04:33:54.788758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.788739 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f61929c-96a1-4073-8f8c-acb25ad6d040-kube-api-access-btr5v\" (UniqueName: \"kubernetes.io/projected/0f61929c-96a1-4073-8f8c-acb25ad6d040-kube-api-access-btr5v\") pod \"0f61929c-96a1-4073-8f8c-acb25ad6d040\" (UID: \"0f61929c-96a1-4073-8f8c-acb25ad6d040\") " Oct 30 04:33:54.789357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.789303 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f61929c-96a1-4073-8f8c-acb25ad6d040-plugins-config-volume" pod "0f61929c-96a1-4073-8f8c-acb25ad6d040" (UID: "0f61929c-96a1-4073-8f8c-acb25ad6d040"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:33:54.789726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.789699 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f61929c-96a1-4073-8f8c-acb25ad6d040-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:54.791764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.791735 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f61929c-96a1-4073-8f8c-acb25ad6d040-kube-api-access-btr5v" pod "0f61929c-96a1-4073-8f8c-acb25ad6d040" (UID: "0f61929c-96a1-4073-8f8c-acb25ad6d040"). InnerVolumeSpecName "kube-api-access-btr5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:33:54.878760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.878726 4793 generic.go:356] "Generic (PLEG): container finished" podID="0f61929c-96a1-4073-8f8c-acb25ad6d040" containerID="ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51" exitCode=0 Oct 30 04:33:54.878987 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.878788 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" Oct 30 04:33:54.878987 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.878809 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" event={"ID":"0f61929c-96a1-4073-8f8c-acb25ad6d040","Type":"ContainerDied","Data":"ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51"} Oct 30 04:33:54.878987 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.878879 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt" event={"ID":"0f61929c-96a1-4073-8f8c-acb25ad6d040","Type":"ContainerDied","Data":"a9b7bf90c244324807e6807a3f307eb73998bc9c31302e746753bf0e3cac2ea9"} Oct 30 04:33:54.878987 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.878911 4793 scope.go:122] "RemoveContainer" containerID="ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51" Oct 30 04:33:54.890630 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.890601 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:54.890846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.890825 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btr5v\" (UniqueName: \"kubernetes.io/projected/0f61929c-96a1-4073-8f8c-acb25ad6d040-kube-api-access-btr5v\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:54.891831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.891814 4793 scope.go:122] "RemoveContainer" containerID="ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51" Oct 30 04:33:54.892133 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:33:54.892112 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51\": container with ID starting with ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51 not found: ID does not exist" containerID="ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51" Oct 30 04:33:54.892194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.892144 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51"} err="failed to get container status \"ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51\": rpc error: code = NotFound desc = could not find container \"ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51\": container with ID starting with ad4478a69437d0005e6f71350380e94461a53dd181d3c50060944a1525336a51 not found: ID does not exist" Oct 30 04:33:54.934752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.934729 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt"] Oct 30 04:33:54.942303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:54.942280 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7a36d-79d85f95f-qtvjt"] Oct 30 04:33:55.898565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:55.898534 4793 generic.go:356] "Generic (PLEG): container finished" podID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerID="1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63" exitCode=0 Oct 30 04:33:55.898977 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:55.898615 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" event={"ID":"ac8095c1-91a7-4082-84d2-e52b612e736b","Type":"ContainerDied","Data":"1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63"} Oct 30 04:33:55.899222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:55.899201 4793 scope.go:122] "RemoveContainer" containerID="1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63" Oct 30 04:33:55.900764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:55.900734 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:55.900881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:55.900805 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:55.900881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:55.900839 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:33:55.922267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:55.922243 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:55.934919 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:55.934897 4793 patch_prober.go:29] interesting pod/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:33:55.935004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:55.934940 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:33:56.227206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.227164 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:56.396372 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.396337 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:56.486089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.486012 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6"] Oct 30 04:33:56.486616 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.486599 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0f61929c-96a1-4073-8f8c-acb25ad6d040" containerName="payload-pre-processing" Oct 30 04:33:56.486616 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.486614 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f61929c-96a1-4073-8f8c-acb25ad6d040" containerName="payload-pre-processing" Oct 30 04:33:56.486787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.486728 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f61929c-96a1-4073-8f8c-acb25ad6d040" containerName="payload-pre-processing" Oct 30 04:33:56.489284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.489256 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" Oct 30 04:33:56.506063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.506038 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2696d8-dockercfg-jr5vv\"" Oct 30 04:33:56.507498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.507280 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2696d8\"" Oct 30 04:33:56.512955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.512931 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6"] Oct 30 04:33:56.669438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.643952 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:56.703257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.703219 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h"] Oct 30 04:33:56.707549 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.707525 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" Oct 30 04:33:56.709052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.709021 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h"] Oct 30 04:33:56.719325 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.719299 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z868\" (UniqueName: \"kubernetes.io/projected/c8223981-1aa9-439f-bd78-f7b2edeaaa29-kube-api-access-2z868\") pod \"payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6\" (UID: \"c8223981-1aa9-439f-bd78-f7b2edeaaa29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" Oct 30 04:33:56.719476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.719344 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8223981-1aa9-439f-bd78-f7b2edeaaa29-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6\" (UID: \"c8223981-1aa9-439f-bd78-f7b2edeaaa29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" Oct 30 04:33:56.719476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.719382 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267efa5e-64d0-41a1-a43e-28b81ed93bd9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h\" (UID: \"267efa5e-64d0-41a1-a43e-28b81ed93bd9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" Oct 30 04:33:56.719476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.719405 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqwrw\" (UniqueName: \"kubernetes.io/projected/267efa5e-64d0-41a1-a43e-28b81ed93bd9-kube-api-access-qqwrw\") pod \"payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h\" (UID: \"267efa5e-64d0-41a1-a43e-28b81ed93bd9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" Oct 30 04:33:56.823852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.820824 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z868\" (UniqueName: \"kubernetes.io/projected/c8223981-1aa9-439f-bd78-f7b2edeaaa29-kube-api-access-2z868\") pod \"payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6\" (UID: \"c8223981-1aa9-439f-bd78-f7b2edeaaa29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" Oct 30 04:33:56.823852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.820912 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8223981-1aa9-439f-bd78-f7b2edeaaa29-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6\" (UID: \"c8223981-1aa9-439f-bd78-f7b2edeaaa29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" Oct 30 04:33:56.823852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.820980 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267efa5e-64d0-41a1-a43e-28b81ed93bd9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h\" (UID: \"267efa5e-64d0-41a1-a43e-28b81ed93bd9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" Oct 30 04:33:56.823852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.821062 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqwrw\" (UniqueName: \"kubernetes.io/projected/267efa5e-64d0-41a1-a43e-28b81ed93bd9-kube-api-access-qqwrw\") pod \"payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h\" (UID: \"267efa5e-64d0-41a1-a43e-28b81ed93bd9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" Oct 30 04:33:56.838651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.838620 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-2696d8\"" Oct 30 04:33:56.842842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.842814 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8223981-1aa9-439f-bd78-f7b2edeaaa29-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6\" (UID: \"c8223981-1aa9-439f-bd78-f7b2edeaaa29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" Oct 30 04:33:56.843053 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.842814 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267efa5e-64d0-41a1-a43e-28b81ed93bd9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h\" (UID: \"267efa5e-64d0-41a1-a43e-28b81ed93bd9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" Oct 30 04:33:56.845323 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.845301 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqwrw\" (UniqueName: \"kubernetes.io/projected/267efa5e-64d0-41a1-a43e-28b81ed93bd9-kube-api-access-qqwrw\") pod \"payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h\" (UID: \"267efa5e-64d0-41a1-a43e-28b81ed93bd9\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" Oct 30 04:33:56.845443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.845403 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z868\" (UniqueName: \"kubernetes.io/projected/c8223981-1aa9-439f-bd78-f7b2edeaaa29-kube-api-access-2z868\") pod \"payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6\" (UID: \"c8223981-1aa9-439f-bd78-f7b2edeaaa29\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" Oct 30 04:33:56.847428 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.847392 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f61929c-96a1-4073-8f8c-acb25ad6d040" path="/var/lib/kubelet/pods/0f61929c-96a1-4073-8f8c-acb25ad6d040/volumes" Oct 30 04:33:56.909255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:56.909221 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" event={"ID":"ac8095c1-91a7-4082-84d2-e52b612e736b","Type":"ContainerStarted","Data":"ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e"} Oct 30 04:33:57.039774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.039741 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-2696d8-dockercfg-jr5vv\"" Oct 30 04:33:57.045886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.045862 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" Oct 30 04:33:57.102447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.102401 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" Oct 30 04:33:57.410263 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.410234 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h"] Oct 30 04:33:57.415314 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:33:57.415288 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267efa5e_64d0_41a1_a43e_28b81ed93bd9.slice/crio-ff02922dbfc26ee4c38791b6741a0dc925017610273385fc08eda3795e587e8d WatchSource:0}: Error finding container ff02922dbfc26ee4c38791b6741a0dc925017610273385fc08eda3795e587e8d: Status 404 returned error can't find the container with id ff02922dbfc26ee4c38791b6741a0dc925017610273385fc08eda3795e587e8d Oct 30 04:33:57.486480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.485269 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6"] Oct 30 04:33:57.505900 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:33:57.505870 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8223981_1aa9_439f_bd78_f7b2edeaaa29.slice/crio-e34243aa06af527fe9b896ad8bc1d5cdfb603a2e2edf290da81899292975bfd8 WatchSource:0}: Error finding container e34243aa06af527fe9b896ad8bc1d5cdfb603a2e2edf290da81899292975bfd8: Status 404 returned error can't find the container with id e34243aa06af527fe9b896ad8bc1d5cdfb603a2e2edf290da81899292975bfd8 Oct 30 04:33:57.718437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.715060 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:57.804914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.804893 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:33:57.836702 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.836675 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-podinfo\") pod \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " Oct 30 04:33:57.836839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.836770 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-envoy\") pod \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " Oct 30 04:33:57.836839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.836819 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-certs\") pod \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " Oct 30 04:33:57.836937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.836879 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-data\") pod \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " Oct 30 04:33:57.836937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.836927 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-kube-api-access-nzk85\" (UniqueName: \"kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-kube-api-access-nzk85\") pod \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " Oct 30 04:33:57.837062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.836958 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-token\" (UniqueName: \"kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-token\") pod \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " Oct 30 04:33:57.837062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837000 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-socket\") pod \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " Oct 30 04:33:57.837062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837052 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-credential-socket\") pod \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " Oct 30 04:33:57.837220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837164 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istiod-ca-cert\") pod \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\" (UID: \"40b57c75-ef43-4572-9e9c-2c56102a6d5b\") " Oct 30 04:33:57.837220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837173 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-certs" pod "40b57c75-ef43-4572-9e9c-2c56102a6d5b" (UID: "40b57c75-ef43-4572-9e9c-2c56102a6d5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:33:57.837220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837202 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-data" pod "40b57c75-ef43-4572-9e9c-2c56102a6d5b" (UID: "40b57c75-ef43-4572-9e9c-2c56102a6d5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:33:57.837407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837383 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-credential-socket" pod "40b57c75-ef43-4572-9e9c-2c56102a6d5b" (UID: "40b57c75-ef43-4572-9e9c-2c56102a6d5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:33:57.837506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837398 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-socket" pod "40b57c75-ef43-4572-9e9c-2c56102a6d5b" (UID: "40b57c75-ef43-4572-9e9c-2c56102a6d5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:33:57.838481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837661 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istiod-ca-cert" pod "40b57c75-ef43-4572-9e9c-2c56102a6d5b" (UID: "40b57c75-ef43-4572-9e9c-2c56102a6d5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:33:57.838481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837774 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:57.838481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837791 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:57.838481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837800 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:57.838481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837807 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:57.838481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.837815 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:57.839503 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.839472 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-podinfo" pod "40b57c75-ef43-4572-9e9c-2c56102a6d5b" (UID: "40b57c75-ef43-4572-9e9c-2c56102a6d5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:33:57.839740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.839712 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-envoy" pod "40b57c75-ef43-4572-9e9c-2c56102a6d5b" (UID: "40b57c75-ef43-4572-9e9c-2c56102a6d5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:33:57.839896 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.839875 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-kube-api-access-nzk85" pod "40b57c75-ef43-4572-9e9c-2c56102a6d5b" (UID: "40b57c75-ef43-4572-9e9c-2c56102a6d5b"). InnerVolumeSpecName "kube-api-access-nzk85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:33:57.839955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.839930 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-token" pod "40b57c75-ef43-4572-9e9c-2c56102a6d5b" (UID: "40b57c75-ef43-4572-9e9c-2c56102a6d5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:33:57.908317 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.908266 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:33:57.909641 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.909307 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 04:33:57.909641 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.909374 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 04:33:57.920189 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.920157 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" event={"ID":"c8223981-1aa9-439f-bd78-f7b2edeaaa29","Type":"ContainerStarted","Data":"ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710"} Oct 30 04:33:57.920314 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.920219 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" Oct 30 04:33:57.920314 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.920243 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" event={"ID":"c8223981-1aa9-439f-bd78-f7b2edeaaa29","Type":"ContainerStarted","Data":"e34243aa06af527fe9b896ad8bc1d5cdfb603a2e2edf290da81899292975bfd8"} Oct 30 04:33:57.921975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.921951 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" event={"ID":"267efa5e-64d0-41a1-a43e-28b81ed93bd9","Type":"ContainerStarted","Data":"b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079"} Oct 30 04:33:57.922089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.922006 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" Oct 30 04:33:57.922089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.922032 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" event={"ID":"267efa5e-64d0-41a1-a43e-28b81ed93bd9","Type":"ContainerStarted","Data":"ff02922dbfc26ee4c38791b6741a0dc925017610273385fc08eda3795e587e8d"} Oct 30 04:33:57.923792 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.923769 4793 generic.go:356] "Generic (PLEG): container finished" podID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerID="068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2" exitCode=0 Oct 30 04:33:57.923914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.923810 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" Oct 30 04:33:57.923914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.923839 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" event={"ID":"40b57c75-ef43-4572-9e9c-2c56102a6d5b","Type":"ContainerDied","Data":"068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2"} Oct 30 04:33:57.923914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.923871 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm" event={"ID":"40b57c75-ef43-4572-9e9c-2c56102a6d5b","Type":"ContainerDied","Data":"5f2d36a121c0fc6ae69c9fd15b39778608286f688bab7df6324f460f0f0d4728"} Oct 30 04:33:57.923914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.923892 4793 scope.go:122] "RemoveContainer" containerID="068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2" Oct 30 04:33:57.935288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.935269 4793 scope.go:122] "RemoveContainer" containerID="e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1" Oct 30 04:33:57.939681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.939657 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:57.939779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.939686 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:57.939779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.939704 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzk85\" (UniqueName: \"kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-kube-api-access-nzk85\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:57.939779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.939720 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40b57c75-ef43-4572-9e9c-2c56102a6d5b-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:33:57.945141 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.945124 4793 scope.go:122] "RemoveContainer" containerID="068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2" Oct 30 04:33:57.945394 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:33:57.945374 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2\": container with ID starting with 068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2 not found: ID does not exist" containerID="068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2" Oct 30 04:33:57.945520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.945399 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2"} err="failed to get container status \"068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2\": rpc error: code = NotFound desc = could not find container \"068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2\": container with ID starting with 068abc0f305d1e53abaa89f3a7fa63d2c2cc1256d11a28d7d254c3bfb19422c2 not found: ID does not exist" Oct 30 04:33:57.945520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.945447 4793 scope.go:122] "RemoveContainer" containerID="e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1" Oct 30 04:33:57.945682 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:33:57.945664 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1\": container with ID starting with e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1 not found: ID does not exist" containerID="e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1" Oct 30 04:33:57.945729 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:57.945687 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1"} err="failed to get container status \"e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1\": rpc error: code = NotFound desc = could not find container \"e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1\": container with ID starting with e17287b96f95dc4c93875a0fea72e74dcaa1eea4d33e76e73890d462544904b1 not found: ID does not exist" Oct 30 04:33:58.004729 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:58.004627 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" podStartSLOduration=2.004609673 podStartE2EDuration="2.004609673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:57.999301978 +0000 UTC m=+2161.285176753" watchObservedRunningTime="2026-10-30 04:33:58.004609673 +0000 UTC m=+2161.290484448" Oct 30 04:33:58.078635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:58.076698 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" podStartSLOduration=2.076679661 podStartE2EDuration="2.076679661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:33:58.074991707 +0000 UTC m=+2161.360866484" watchObservedRunningTime="2026-10-30 04:33:58.076679661 +0000 UTC m=+2161.362554436" Oct 30 04:33:58.117004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:58.116972 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm"] Oct 30 04:33:58.123486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:58.123461 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7a36d-openshift-default-5756486b57-pgrdm"] Oct 30 04:33:58.889321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:58.889287 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" path="/var/lib/kubelet/pods/40b57c75-ef43-4572-9e9c-2c56102a6d5b/volumes" Oct 30 04:33:58.907826 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:58.907793 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 04:33:58.907983 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:58.907864 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 04:33:59.545749 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:59.545712 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:59.790390 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:59.790350 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:33:59.908151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:59.908072 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 04:33:59.908151 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:33:59.908122 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 04:34:00.908356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:00.908316 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 04:34:00.908754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:00.908388 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 04:34:01.907694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:01.907660 4793 patch_prober.go:29] interesting pod/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 04:34:01.907877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:01.907719 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 04:34:02.014452 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:02.013550 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:02.912188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:02.912158 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:34:02.912648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:02.912627 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:34:02.913287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:02.913271 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:34:03.702848 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:03.702806 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:03.930250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:03.930215 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:05.358442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:05.355392 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:05.579704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:05.579662 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:07.089810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.089768 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:07.602705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.602673 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh"] Oct 30 04:34:07.603296 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.603275 4793 cpu_manager.go:414] "containerMap: removing container" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" Oct 30 04:34:07.603400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.603298 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" Oct 30 04:34:07.603400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.603347 4793 cpu_manager.go:414] "containerMap: removing container" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" Oct 30 04:34:07.603400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.603356 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" Oct 30 04:34:07.603587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.603549 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" Oct 30 04:34:07.603587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.603571 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b57c75-ef43-4572-9e9c-2c56102a6d5b" containerName="istio-proxy" Oct 30 04:34:07.607564 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.607541 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:07.624447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.624411 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh"] Oct 30 04:34:07.633073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.633048 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:34:07.633202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.633078 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:34:07.635568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.635535 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:34:07.635666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.635596 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:34:07.711814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.711785 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afcc50b1-5hkrh\" (UID: \"13d97113-2871-4d8b-a2ba-8b6cb0272160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:07.711935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.711830 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afcc50b1-5hkrh\" (UID: \"13d97113-2871-4d8b-a2ba-8b6cb0272160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:07.712003 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.711984 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtsqp\" (UniqueName: \"kubernetes.io/projected/13d97113-2871-4d8b-a2ba-8b6cb0272160-kube-api-access-jtsqp\") pod \"maas-api-revoke-sub-afcc50b1-5hkrh\" (UID: \"13d97113-2871-4d8b-a2ba-8b6cb0272160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:07.813655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.813615 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtsqp\" (UniqueName: \"kubernetes.io/projected/13d97113-2871-4d8b-a2ba-8b6cb0272160-kube-api-access-jtsqp\") pod \"maas-api-revoke-sub-afcc50b1-5hkrh\" (UID: \"13d97113-2871-4d8b-a2ba-8b6cb0272160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:07.813867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.813834 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afcc50b1-5hkrh\" (UID: \"13d97113-2871-4d8b-a2ba-8b6cb0272160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:07.813965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.813885 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afcc50b1-5hkrh\" (UID: \"13d97113-2871-4d8b-a2ba-8b6cb0272160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:07.814632 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.814592 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afcc50b1-5hkrh\" (UID: \"13d97113-2871-4d8b-a2ba-8b6cb0272160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:07.816451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.816414 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afcc50b1-5hkrh\" (UID: \"13d97113-2871-4d8b-a2ba-8b6cb0272160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:07.837072 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.837050 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtsqp\" (UniqueName: \"kubernetes.io/projected/13d97113-2871-4d8b-a2ba-8b6cb0272160-kube-api-access-jtsqp\") pod \"maas-api-revoke-sub-afcc50b1-5hkrh\" (UID: \"13d97113-2871-4d8b-a2ba-8b6cb0272160\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:07.920140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:07.920077 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:08.230585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:08.230541 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh"] Oct 30 04:34:08.233701 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:34:08.233671 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d97113_2871_4d8b_a2ba_8b6cb0272160.slice/crio-90ea765aac01c1afe4a179d0a06f8e9c1964102a57ee814ca2f899ab693984b5 WatchSource:0}: Error finding container 90ea765aac01c1afe4a179d0a06f8e9c1964102a57ee814ca2f899ab693984b5: Status 404 returned error can't find the container with id 90ea765aac01c1afe4a179d0a06f8e9c1964102a57ee814ca2f899ab693984b5 Oct 30 04:34:08.250907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:08.250874 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:08.789873 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:08.789836 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:08.831850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:08.831768 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct"] Oct 30 04:34:08.933795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:08.933767 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" Oct 30 04:34:08.934301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:08.934281 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" Oct 30 04:34:09.035363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:09.035334 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" event={"ID":"13d97113-2871-4d8b-a2ba-8b6cb0272160","Type":"ContainerStarted","Data":"4548450670fa751210dda46e0f962a945e74cd9cbd20dd8ea1e6b209858ccb1c"} Oct 30 04:34:09.035562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:09.035386 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" event={"ID":"13d97113-2871-4d8b-a2ba-8b6cb0272160","Type":"ContainerStarted","Data":"90ea765aac01c1afe4a179d0a06f8e9c1964102a57ee814ca2f899ab693984b5"} Oct 30 04:34:09.168758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:09.168362 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" podStartSLOduration=2.168343056 podStartE2EDuration="2.168343056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:34:09.166366127 +0000 UTC m=+2172.452240929" watchObservedRunningTime="2026-10-30 04:34:09.168343056 +0000 UTC m=+2172.454217835" Oct 30 04:34:09.230678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:09.230641 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:09.487918 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:09.487877 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:10.043844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:10.043811 4793 generic.go:356] "Generic (PLEG): container finished" podID="13d97113-2871-4d8b-a2ba-8b6cb0272160" containerID="4548450670fa751210dda46e0f962a945e74cd9cbd20dd8ea1e6b209858ccb1c" exitCode=0 Oct 30 04:34:10.044014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:10.043892 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" event={"ID":"13d97113-2871-4d8b-a2ba-8b6cb0272160","Type":"ContainerDied","Data":"4548450670fa751210dda46e0f962a945e74cd9cbd20dd8ea1e6b209858ccb1c"} Oct 30 04:34:10.729240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:10.729208 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:11.219167 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:34:11.219134 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:34:11.225198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:11.225175 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:11.364895 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:11.364820 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-openshift-ca\") pod \"13d97113-2871-4d8b-a2ba-8b6cb0272160\" (UID: \"13d97113-2871-4d8b-a2ba-8b6cb0272160\") " Oct 30 04:34:11.365030 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:11.364937 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-xks-ca\") pod \"13d97113-2871-4d8b-a2ba-8b6cb0272160\" (UID: \"13d97113-2871-4d8b-a2ba-8b6cb0272160\") " Oct 30 04:34:11.365030 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:11.364986 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13d97113-2871-4d8b-a2ba-8b6cb0272160-kube-api-access-jtsqp\" (UniqueName: \"kubernetes.io/projected/13d97113-2871-4d8b-a2ba-8b6cb0272160-kube-api-access-jtsqp\") pod \"13d97113-2871-4d8b-a2ba-8b6cb0272160\" (UID: \"13d97113-2871-4d8b-a2ba-8b6cb0272160\") " Oct 30 04:34:11.365248 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:11.365216 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-openshift-ca" pod "13d97113-2871-4d8b-a2ba-8b6cb0272160" (UID: "13d97113-2871-4d8b-a2ba-8b6cb0272160"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:34:11.365434 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:11.365400 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:34:11.367046 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:11.367020 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-xks-ca" pod "13d97113-2871-4d8b-a2ba-8b6cb0272160" (UID: "13d97113-2871-4d8b-a2ba-8b6cb0272160"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:34:11.367133 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:11.367108 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d97113-2871-4d8b-a2ba-8b6cb0272160-kube-api-access-jtsqp" pod "13d97113-2871-4d8b-a2ba-8b6cb0272160" (UID: "13d97113-2871-4d8b-a2ba-8b6cb0272160"). InnerVolumeSpecName "kube-api-access-jtsqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:34:11.466851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:11.466826 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtsqp\" (UniqueName: \"kubernetes.io/projected/13d97113-2871-4d8b-a2ba-8b6cb0272160-kube-api-access-jtsqp\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:34:11.466968 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:11.466855 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13d97113-2871-4d8b-a2ba-8b6cb0272160-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:34:12.060461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.060409 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" Oct 30 04:34:12.060855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.060462 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh" event={"ID":"13d97113-2871-4d8b-a2ba-8b6cb0272160","Type":"ContainerDied","Data":"90ea765aac01c1afe4a179d0a06f8e9c1964102a57ee814ca2f899ab693984b5"} Oct 30 04:34:12.060855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.060519 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90ea765aac01c1afe4a179d0a06f8e9c1964102a57ee814ca2f899ab693984b5" Oct 30 04:34:12.186932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.186899 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn"] Oct 30 04:34:12.187553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.187534 4793 cpu_manager.go:414] "containerMap: removing container" podUID="13d97113-2871-4d8b-a2ba-8b6cb0272160" containerName="revoke-keys" Oct 30 04:34:12.187672 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.187562 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13d97113-2871-4d8b-a2ba-8b6cb0272160" containerName="revoke-keys" Oct 30 04:34:12.187764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.187750 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="13d97113-2871-4d8b-a2ba-8b6cb0272160" containerName="revoke-keys" Oct 30 04:34:12.190546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.190526 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.206648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.206621 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:34:12.210506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.210483 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn"] Oct 30 04:34:12.221672 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.221634 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-2696d8-serving-cert\"" Oct 30 04:34:12.221804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.221702 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gw5bk\"" Oct 30 04:34:12.221804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.221757 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 04:34:12.221930 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.221835 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:34:12.410520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.407289 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5lfl\" (UniqueName: \"kubernetes.io/projected/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-kube-api-access-m5lfl\") pod \"maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.410520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.407398 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.410520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.407443 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.410520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.408019 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-metrics-certs\") pod \"maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.410520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.408834 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:12.509693 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.509659 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.509894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.509696 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.509894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.509766 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-metrics-certs\") pod \"maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.510019 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.509964 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5lfl\" (UniqueName: \"kubernetes.io/projected/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-kube-api-access-m5lfl\") pod \"maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.510663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.510633 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.512795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.512775 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.512895 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.512816 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-metrics-certs\") pod \"maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.534721 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.534698 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5lfl\" (UniqueName: \"kubernetes.io/projected/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-kube-api-access-m5lfl\") pod \"maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.636218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.636188 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:12.804246 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.804212 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:12.844119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:12.844083 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:13.111385 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:34:13.111356 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e1dbe7_9bd2_49c3_9ba9_6db2fdf6e7e2.slice/crio-592a528bea9b546c19bbe667580c9fc2c8fd6dfe94250a4ab3fb0199bfa83e1e WatchSource:0}: Error finding container 592a528bea9b546c19bbe667580c9fc2c8fd6dfe94250a4ab3fb0199bfa83e1e: Status 404 returned error can't find the container with id 592a528bea9b546c19bbe667580c9fc2c8fd6dfe94250a4ab3fb0199bfa83e1e Oct 30 04:34:13.111726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:13.111570 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn"] Oct 30 04:34:13.112824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:13.112806 4793 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:34:13.832893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:13.832806 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" podUID="73e08ad2-cda0-4914-b839-081b2f791a82" containerName="payload-processing" containerID="cri-o://82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2" gracePeriod=25 Oct 30 04:34:14.086009 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:14.084776 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" event={"ID":"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2","Type":"ContainerStarted","Data":"6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0"} Oct 30 04:34:14.086009 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:14.084853 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" event={"ID":"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2","Type":"ContainerStarted","Data":"592a528bea9b546c19bbe667580c9fc2c8fd6dfe94250a4ab3fb0199bfa83e1e"} Oct 30 04:34:14.086009 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:14.085941 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:14.130514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:14.130465 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" podStartSLOduration=1.798499566 podStartE2EDuration="2.13045157s" totalImagesPullingTime="331.952004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:34:14.12904441 +0000 UTC m=+2177.414919185" watchObservedRunningTime="2026-10-30 04:34:14.13045157 +0000 UTC m=+2177.416326338" Oct 30 04:34:14.133688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:14.133654 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:15.491819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:15.491783 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:15.847945 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:15.847862 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:17.529389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:17.529351 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:19.054032 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:19.053991 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:19.284892 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:19.284854 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:19.502527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:19.502493 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:20.168561 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:20.168520 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:21.112732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:21.112705 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:34:21.192878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:21.192835 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" podUID="73e08ad2-cda0-4914-b839-081b2f791a82" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:34:21.258689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:21.258655 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:21.485056 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:21.485006 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:22.009134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:22.009100 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:22.251653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:22.251619 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:22.479468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:22.479433 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:22.918852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:22.918749 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:23.361538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:23.361480 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:23.943596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:23.943556 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:24.176004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:24.175963 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:25.939288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:25.939246 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:27.868978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:27.868929 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:29.616525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:29.616485 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:29.956320 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:29.956285 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:31.159331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:31.159295 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:31.192404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:31.192371 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" podUID="73e08ad2-cda0-4914-b839-081b2f791a82" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 04:34:31.820805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:31.820764 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:33.512841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:33.512806 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:33.823180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:33.823105 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn"] Oct 30 04:34:33.827219 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:33.827202 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:33.845837 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:33.845809 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn"] Oct 30 04:34:33.889959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:33.889927 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:34:33.890095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:33.890007 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:34:33.944687 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:33.944376 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvqhk\" (UniqueName: \"kubernetes.io/projected/83183572-0667-4816-88b4-4f85b8aa041a-kube-api-access-pvqhk\") pod \"maas-api-revoke-sub-44ed8a9f-njvnn\" (UID: \"83183572-0667-4816-88b4-4f85b8aa041a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:33.944687 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:33.944485 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44ed8a9f-njvnn\" (UID: \"83183572-0667-4816-88b4-4f85b8aa041a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:33.944687 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:33.944676 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44ed8a9f-njvnn\" (UID: \"83183572-0667-4816-88b4-4f85b8aa041a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:34.046604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.046571 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvqhk\" (UniqueName: \"kubernetes.io/projected/83183572-0667-4816-88b4-4f85b8aa041a-kube-api-access-pvqhk\") pod \"maas-api-revoke-sub-44ed8a9f-njvnn\" (UID: \"83183572-0667-4816-88b4-4f85b8aa041a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:34.046758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.046620 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44ed8a9f-njvnn\" (UID: \"83183572-0667-4816-88b4-4f85b8aa041a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:34.046827 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.046757 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44ed8a9f-njvnn\" (UID: \"83183572-0667-4816-88b4-4f85b8aa041a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:34.048590 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.048567 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44ed8a9f-njvnn\" (UID: \"83183572-0667-4816-88b4-4f85b8aa041a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:34.051062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.051038 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44ed8a9f-njvnn\" (UID: \"83183572-0667-4816-88b4-4f85b8aa041a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:34.097326 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.097252 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvqhk\" (UniqueName: \"kubernetes.io/projected/83183572-0667-4816-88b4-4f85b8aa041a-kube-api-access-pvqhk\") pod \"maas-api-revoke-sub-44ed8a9f-njvnn\" (UID: \"83183572-0667-4816-88b4-4f85b8aa041a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:34.142587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.142553 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:34.207610 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.207583 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" Oct 30 04:34:34.281440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.278459 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e08ad2-cda0-4914-b839-081b2f791a82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e08ad2-cda0-4914-b839-081b2f791a82-plugins-config-volume\") pod \"73e08ad2-cda0-4914-b839-081b2f791a82\" (UID: \"73e08ad2-cda0-4914-b839-081b2f791a82\") " Oct 30 04:34:34.281440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.278522 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e08ad2-cda0-4914-b839-081b2f791a82-kube-api-access-g8bwr\" (UniqueName: \"kubernetes.io/projected/73e08ad2-cda0-4914-b839-081b2f791a82-kube-api-access-g8bwr\") pod \"73e08ad2-cda0-4914-b839-081b2f791a82\" (UID: \"73e08ad2-cda0-4914-b839-081b2f791a82\") " Oct 30 04:34:34.296469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.295631 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e08ad2-cda0-4914-b839-081b2f791a82-plugins-config-volume" pod "73e08ad2-cda0-4914-b839-081b2f791a82" (UID: "73e08ad2-cda0-4914-b839-081b2f791a82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:34:34.301341 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.301271 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e08ad2-cda0-4914-b839-081b2f791a82-kube-api-access-g8bwr" pod "73e08ad2-cda0-4914-b839-081b2f791a82" (UID: "73e08ad2-cda0-4914-b839-081b2f791a82"). InnerVolumeSpecName "kube-api-access-g8bwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:34:34.306394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.306361 4793 generic.go:356] "Generic (PLEG): container finished" podID="73e08ad2-cda0-4914-b839-081b2f791a82" containerID="82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2" exitCode=0 Oct 30 04:34:34.306558 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.306508 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" event={"ID":"73e08ad2-cda0-4914-b839-081b2f791a82","Type":"ContainerDied","Data":"82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2"} Oct 30 04:34:34.306624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.306568 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" event={"ID":"73e08ad2-cda0-4914-b839-081b2f791a82","Type":"ContainerDied","Data":"996214d39ac0af72e1883348985992dd33ac7875f1f79583768ed211441b46f3"} Oct 30 04:34:34.306624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.306604 4793 scope.go:122] "RemoveContainer" containerID="82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2" Oct 30 04:34:34.306790 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.306772 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct" Oct 30 04:34:34.320166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.320106 4793 scope.go:122] "RemoveContainer" containerID="82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2" Oct 30 04:34:34.320529 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:34:34.320470 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2\": container with ID starting with 82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2 not found: ID does not exist" containerID="82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2" Oct 30 04:34:34.320529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.320507 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2"} err="failed to get container status \"82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2\": rpc error: code = NotFound desc = could not find container \"82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2\": container with ID starting with 82e455087b94c1a830a039bcce80feefe885ba909ee5d03f5ae068e24d1e67b2 not found: ID does not exist" Oct 30 04:34:34.395277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.394530 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct"] Oct 30 04:34:34.395917 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.395889 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73e08ad2-cda0-4914-b839-081b2f791a82-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:34:34.395917 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.395918 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8bwr\" (UniqueName: \"kubernetes.io/projected/73e08ad2-cda0-4914-b839-081b2f791a82-kube-api-access-g8bwr\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:34:34.400864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.400842 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-153a44-969747556-xr6ct"] Oct 30 04:34:34.494871 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:34:34.494826 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83183572_0667_4816_88b4_4f85b8aa041a.slice/crio-8dbaa26d285ecfe20a7e2eeae944f91f60bc4397a8f1b851ba8d81675686cff6 WatchSource:0}: Error finding container 8dbaa26d285ecfe20a7e2eeae944f91f60bc4397a8f1b851ba8d81675686cff6: Status 404 returned error can't find the container with id 8dbaa26d285ecfe20a7e2eeae944f91f60bc4397a8f1b851ba8d81675686cff6 Oct 30 04:34:34.495382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.495355 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn"] Oct 30 04:34:34.847227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:34.847183 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e08ad2-cda0-4914-b839-081b2f791a82" path="/var/lib/kubelet/pods/73e08ad2-cda0-4914-b839-081b2f791a82/volumes" Oct 30 04:34:35.308226 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:35.308174 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:35.317523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:35.317484 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" event={"ID":"83183572-0667-4816-88b4-4f85b8aa041a","Type":"ContainerStarted","Data":"db2edba6f3e429f1e125a93020cf826c1fcc3799584b39e54bc6f2000c610299"} Oct 30 04:34:35.317666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:35.317548 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" event={"ID":"83183572-0667-4816-88b4-4f85b8aa041a","Type":"ContainerStarted","Data":"8dbaa26d285ecfe20a7e2eeae944f91f60bc4397a8f1b851ba8d81675686cff6"} Oct 30 04:34:35.530829 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:35.530784 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:36.328240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:36.327943 4793 generic.go:356] "Generic (PLEG): container finished" podID="83183572-0667-4816-88b4-4f85b8aa041a" containerID="db2edba6f3e429f1e125a93020cf826c1fcc3799584b39e54bc6f2000c610299" exitCode=0 Oct 30 04:34:36.328240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:36.327990 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" event={"ID":"83183572-0667-4816-88b4-4f85b8aa041a","Type":"ContainerDied","Data":"db2edba6f3e429f1e125a93020cf826c1fcc3799584b39e54bc6f2000c610299"} Oct 30 04:34:36.906361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:36.906332 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:37.131638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:37.131549 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:37.358379 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:37.358347 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:37.501636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:37.501612 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:37.639893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:37.639866 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83183572-0667-4816-88b4-4f85b8aa041a-kube-api-access-pvqhk\" (UniqueName: \"kubernetes.io/projected/83183572-0667-4816-88b4-4f85b8aa041a-kube-api-access-pvqhk\") pod \"83183572-0667-4816-88b4-4f85b8aa041a\" (UID: \"83183572-0667-4816-88b4-4f85b8aa041a\") " Oct 30 04:34:37.640070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:37.639950 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-xks-ca\") pod \"83183572-0667-4816-88b4-4f85b8aa041a\" (UID: \"83183572-0667-4816-88b4-4f85b8aa041a\") " Oct 30 04:34:37.640147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:37.640083 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-openshift-ca\") pod \"83183572-0667-4816-88b4-4f85b8aa041a\" (UID: \"83183572-0667-4816-88b4-4f85b8aa041a\") " Oct 30 04:34:37.640460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:37.640406 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-openshift-ca" pod "83183572-0667-4816-88b4-4f85b8aa041a" (UID: "83183572-0667-4816-88b4-4f85b8aa041a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:34:37.640589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:37.640546 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:34:37.642128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:37.642094 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-xks-ca" pod "83183572-0667-4816-88b4-4f85b8aa041a" (UID: "83183572-0667-4816-88b4-4f85b8aa041a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:34:37.642244 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:37.642136 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83183572-0667-4816-88b4-4f85b8aa041a-kube-api-access-pvqhk" pod "83183572-0667-4816-88b4-4f85b8aa041a" (UID: "83183572-0667-4816-88b4-4f85b8aa041a"). InnerVolumeSpecName "kube-api-access-pvqhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:34:37.741812 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:37.741780 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvqhk\" (UniqueName: \"kubernetes.io/projected/83183572-0667-4816-88b4-4f85b8aa041a-kube-api-access-pvqhk\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:34:37.741812 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:37.741810 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83183572-0667-4816-88b4-4f85b8aa041a-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:34:38.347547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:38.347499 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" Oct 30 04:34:38.347729 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:38.347666 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn" event={"ID":"83183572-0667-4816-88b4-4f85b8aa041a","Type":"ContainerDied","Data":"8dbaa26d285ecfe20a7e2eeae944f91f60bc4397a8f1b851ba8d81675686cff6"} Oct 30 04:34:38.347729 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:38.347722 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dbaa26d285ecfe20a7e2eeae944f91f60bc4397a8f1b851ba8d81675686cff6" Oct 30 04:34:38.645186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:38.645102 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:39.908603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:39.908523 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:34:39.909000 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:39.908615 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:34:40.341344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:40.341299 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:41.221310 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:34:41.221269 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:34:41.373097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:41.373063 4793 generic.go:356] "Generic (PLEG): container finished" podID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerID="323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e" exitCode=0 Oct 30 04:34:41.373271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:41.373134 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" event={"ID":"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e","Type":"ContainerDied","Data":"323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e"} Oct 30 04:34:41.373625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:41.373605 4793 scope.go:122] "RemoveContainer" containerID="323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e" Oct 30 04:34:41.375144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:41.375110 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:34:41.375241 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:41.375188 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:34:41.375241 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:41.375227 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:34:41.838331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:41.838301 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:42.424463 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:42.423486 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:42.424463 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:42.424273 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" event={"ID":"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e","Type":"ContainerStarted","Data":"ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc"} Oct 30 04:34:42.916640 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:42.916607 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:34:42.916932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:42.916906 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 04:34:42.916990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:42.916962 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 04:34:43.102105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:43.102066 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt"] Oct 30 04:34:43.138279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:43.138244 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p"] Oct 30 04:34:43.400443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:43.399616 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:43.614240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:43.614202 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:43.917131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:43.917100 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 04:34:43.917391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:43.917368 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 04:34:44.212466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:44.212286 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:44.424472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:44.424412 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:44.917125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:44.917090 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 04:34:44.917568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:44.917146 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 04:34:45.916989 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:45.916945 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 04:34:45.917229 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:45.917012 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 04:34:45.918791 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:45.918767 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:46.918863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:46.918825 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 04:34:46.919599 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:46.919564 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 04:34:47.640480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:47.639904 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:47.917124 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:47.917038 4793 patch_prober.go:29] interesting pod/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 04:34:47.917124 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:47.917106 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 04:34:48.102689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.102645 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" podUID="53227857-f430-41a9-9a08-3e2daad9f1b7" containerName="payload-processing" containerID="cri-o://57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e" gracePeriod=25 Oct 30 04:34:48.139130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.139093 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" podUID="eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7" containerName="payload-pre-processing" containerID="cri-o://50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee" gracePeriod=25 Oct 30 04:34:48.421625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.421597 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" Oct 30 04:34:48.463936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.463911 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" Oct 30 04:34:48.488328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.488295 4793 generic.go:356] "Generic (PLEG): container finished" podID="eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7" containerID="50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee" exitCode=0 Oct 30 04:34:48.488514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.488339 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" Oct 30 04:34:48.488514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.488375 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" event={"ID":"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7","Type":"ContainerDied","Data":"50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee"} Oct 30 04:34:48.488514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.488481 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p" event={"ID":"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7","Type":"ContainerDied","Data":"b1b8653f258806789d0cc774e0457cb40a2171d19393b77c762c8d56c5b14168"} Oct 30 04:34:48.488679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.488516 4793 scope.go:122] "RemoveContainer" containerID="50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee" Oct 30 04:34:48.489935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.489910 4793 generic.go:356] "Generic (PLEG): container finished" podID="53227857-f430-41a9-9a08-3e2daad9f1b7" containerID="57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e" exitCode=0 Oct 30 04:34:48.490055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.489947 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" Oct 30 04:34:48.490055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.489997 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" event={"ID":"53227857-f430-41a9-9a08-3e2daad9f1b7","Type":"ContainerDied","Data":"57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e"} Oct 30 04:34:48.490152 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.490070 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt" event={"ID":"53227857-f430-41a9-9a08-3e2daad9f1b7","Type":"ContainerDied","Data":"5b36cc4a9b6bd2a95dd59b3dccbce6b0c8c308f938dec3b197fbd8d6449b9a02"} Oct 30 04:34:48.499481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.499462 4793 scope.go:122] "RemoveContainer" containerID="50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee" Oct 30 04:34:48.499808 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:34:48.499781 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee\": container with ID starting with 50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee not found: ID does not exist" containerID="50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee" Oct 30 04:34:48.499912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.499812 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee"} err="failed to get container status \"50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee\": rpc error: code = NotFound desc = could not find container \"50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee\": container with ID starting with 50e60b1f35287c1cda52d2581951f2a2ef0f9e9bf6b83798ccc6566598a284ee not found: ID does not exist" Oct 30 04:34:48.499912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.499834 4793 scope.go:122] "RemoveContainer" containerID="57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e" Oct 30 04:34:48.512719 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.512698 4793 scope.go:122] "RemoveContainer" containerID="57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e" Oct 30 04:34:48.513336 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:34:48.513002 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e\": container with ID starting with 57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e not found: ID does not exist" containerID="57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e" Oct 30 04:34:48.513336 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.513032 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e"} err="failed to get container status \"57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e\": rpc error: code = NotFound desc = could not find container \"57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e\": container with ID starting with 57fd2ddf5bc4107ecf17b46cb8b1cd61e697729c06b5e648c7db86a1919ec66e not found: ID does not exist" Oct 30 04:34:48.536943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.536919 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-plugins-config-volume\") pod \"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7\" (UID: \"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7\") " Oct 30 04:34:48.537054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.536968 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-kube-api-access-2jg9x\" (UniqueName: \"kubernetes.io/projected/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-kube-api-access-2jg9x\") pod \"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7\" (UID: \"eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7\") " Oct 30 04:34:48.537054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.536997 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53227857-f430-41a9-9a08-3e2daad9f1b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53227857-f430-41a9-9a08-3e2daad9f1b7-plugins-config-volume\") pod \"53227857-f430-41a9-9a08-3e2daad9f1b7\" (UID: \"53227857-f430-41a9-9a08-3e2daad9f1b7\") " Oct 30 04:34:48.537166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.537057 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53227857-f430-41a9-9a08-3e2daad9f1b7-kube-api-access-6cw2v\" (UniqueName: \"kubernetes.io/projected/53227857-f430-41a9-9a08-3e2daad9f1b7-kube-api-access-6cw2v\") pod \"53227857-f430-41a9-9a08-3e2daad9f1b7\" (UID: \"53227857-f430-41a9-9a08-3e2daad9f1b7\") " Oct 30 04:34:48.537536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.537515 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53227857-f430-41a9-9a08-3e2daad9f1b7-plugins-config-volume" pod "53227857-f430-41a9-9a08-3e2daad9f1b7" (UID: "53227857-f430-41a9-9a08-3e2daad9f1b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:34:48.537627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.537572 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-plugins-config-volume" pod "eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7" (UID: "eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:34:48.539283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.539260 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53227857-f430-41a9-9a08-3e2daad9f1b7-kube-api-access-6cw2v" pod "53227857-f430-41a9-9a08-3e2daad9f1b7" (UID: "53227857-f430-41a9-9a08-3e2daad9f1b7"). InnerVolumeSpecName "kube-api-access-6cw2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:34:48.539464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.539442 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-kube-api-access-2jg9x" pod "eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7" (UID: "eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7"). InnerVolumeSpecName "kube-api-access-2jg9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:34:48.638897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.638869 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:34:48.638897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.638895 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jg9x\" (UniqueName: \"kubernetes.io/projected/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7-kube-api-access-2jg9x\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:34:48.639057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.638905 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53227857-f430-41a9-9a08-3e2daad9f1b7-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:34:48.639057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.638914 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cw2v\" (UniqueName: \"kubernetes.io/projected/53227857-f430-41a9-9a08-3e2daad9f1b7-kube-api-access-6cw2v\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:34:48.846886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.846858 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt"] Oct 30 04:34:48.852292 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.852267 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-86eaa9-55b6679964-m8kkt"] Oct 30 04:34:48.889411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.889380 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p"] Oct 30 04:34:48.893908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.893884 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-86eaa9-df9d9767-mg98p"] Oct 30 04:34:48.920533 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.920512 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:34:48.921007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.920990 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:34:48.921773 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:48.921757 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:34:49.311587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:49.311540 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:50.218115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:50.218071 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:50.848383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:50.848352 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53227857-f430-41a9-9a08-3e2daad9f1b7" path="/var/lib/kubelet/pods/53227857-f430-41a9-9a08-3e2daad9f1b7/volumes" Oct 30 04:34:50.848790 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:50.848774 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7" path="/var/lib/kubelet/pods/eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7/volumes" Oct 30 04:34:51.204595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:51.204561 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:52.567077 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:52.567042 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:52.811072 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:52.811036 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:53.354621 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:53.354583 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:53.579136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:53.579100 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:53.823087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:53.823034 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:54.468473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:54.468411 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:55.223906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:55.223865 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:56.624859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:56.624823 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:56.854219 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:56.854177 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:57.071840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:57.071807 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:57.356980 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:57.356888 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:57.616111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:57.616030 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x"] Oct 30 04:34:57.616521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:57.616470 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerName="istio-proxy" containerID="cri-o://ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e" gracePeriod=30 Oct 30 04:34:57.767264 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:57.767230 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:58.611966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:58.611872 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:34:58.897768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:34:58.897667 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:00.701149 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:00.701116 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:02.405827 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:02.405789 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:02.909634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:02.909610 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:35:03.016233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.016161 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-podinfo\") pod \"ac8095c1-91a7-4082-84d2-e52b612e736b\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " Oct 30 04:35:03.016389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.016235 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-certs\") pod \"ac8095c1-91a7-4082-84d2-e52b612e736b\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " Oct 30 04:35:03.016389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.016265 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-data\") pod \"ac8095c1-91a7-4082-84d2-e52b612e736b\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " Oct 30 04:35:03.016389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.016312 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac8095c1-91a7-4082-84d2-e52b612e736b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8095c1-91a7-4082-84d2-e52b612e736b-istiod-ca-cert\") pod \"ac8095c1-91a7-4082-84d2-e52b612e736b\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " Oct 30 04:35:03.016389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.016351 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-envoy\") pod \"ac8095c1-91a7-4082-84d2-e52b612e736b\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " Oct 30 04:35:03.016663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.016396 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-token\" (UniqueName: \"kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-token\") pod \"ac8095c1-91a7-4082-84d2-e52b612e736b\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " Oct 30 04:35:03.016663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.016492 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-kube-api-access-bglgf\" (UniqueName: \"kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-kube-api-access-bglgf\") pod \"ac8095c1-91a7-4082-84d2-e52b612e736b\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " Oct 30 04:35:03.016663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.016532 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-credential-socket\") pod \"ac8095c1-91a7-4082-84d2-e52b612e736b\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " Oct 30 04:35:03.016663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.016580 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-socket\") pod \"ac8095c1-91a7-4082-84d2-e52b612e736b\" (UID: \"ac8095c1-91a7-4082-84d2-e52b612e736b\") " Oct 30 04:35:03.016663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.016632 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-data" pod "ac8095c1-91a7-4082-84d2-e52b612e736b" (UID: "ac8095c1-91a7-4082-84d2-e52b612e736b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:03.017110 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.017045 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac8095c1-91a7-4082-84d2-e52b612e736b-istiod-ca-cert" pod "ac8095c1-91a7-4082-84d2-e52b612e736b" (UID: "ac8095c1-91a7-4082-84d2-e52b612e736b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:35:03.017205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.017134 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:03.017205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.017155 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac8095c1-91a7-4082-84d2-e52b612e736b-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:03.017569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.017545 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-credential-socket" pod "ac8095c1-91a7-4082-84d2-e52b612e736b" (UID: "ac8095c1-91a7-4082-84d2-e52b612e736b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:03.017688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.017665 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-socket" pod "ac8095c1-91a7-4082-84d2-e52b612e736b" (UID: "ac8095c1-91a7-4082-84d2-e52b612e736b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:03.017760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.017706 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-certs" pod "ac8095c1-91a7-4082-84d2-e52b612e736b" (UID: "ac8095c1-91a7-4082-84d2-e52b612e736b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:03.018669 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.018636 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-podinfo" pod "ac8095c1-91a7-4082-84d2-e52b612e736b" (UID: "ac8095c1-91a7-4082-84d2-e52b612e736b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:35:03.018810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.018777 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-envoy" pod "ac8095c1-91a7-4082-84d2-e52b612e736b" (UID: "ac8095c1-91a7-4082-84d2-e52b612e736b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:03.019286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.019266 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-token" pod "ac8095c1-91a7-4082-84d2-e52b612e736b" (UID: "ac8095c1-91a7-4082-84d2-e52b612e736b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:35:03.019763 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.019742 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-kube-api-access-bglgf" pod "ac8095c1-91a7-4082-84d2-e52b612e736b" (UID: "ac8095c1-91a7-4082-84d2-e52b612e736b"). InnerVolumeSpecName "kube-api-access-bglgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:35:03.117857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.117826 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:03.117857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.117852 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:03.117857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.117861 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bglgf\" (UniqueName: \"kubernetes.io/projected/ac8095c1-91a7-4082-84d2-e52b612e736b-kube-api-access-bglgf\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:03.118055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.117871 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:03.118055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.117880 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:03.118055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.117889 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac8095c1-91a7-4082-84d2-e52b612e736b-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:03.118055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.117899 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac8095c1-91a7-4082-84d2-e52b612e736b-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:03.615926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.615893 4793 generic.go:356] "Generic (PLEG): container finished" podID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerID="ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e" exitCode=0 Oct 30 04:35:03.616355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.615931 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" event={"ID":"ac8095c1-91a7-4082-84d2-e52b612e736b","Type":"ContainerDied","Data":"ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e"} Oct 30 04:35:03.616355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.615950 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" Oct 30 04:35:03.616355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.615975 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x" event={"ID":"ac8095c1-91a7-4082-84d2-e52b612e736b","Type":"ContainerDied","Data":"6548b5c0a864e16d26e3d6569b0788d01cbbca50463da7bffd622165c453e71d"} Oct 30 04:35:03.616355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.615996 4793 scope.go:122] "RemoveContainer" containerID="ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e" Oct 30 04:35:03.626608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.626588 4793 scope.go:122] "RemoveContainer" containerID="1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63" Oct 30 04:35:03.636198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.636180 4793 scope.go:122] "RemoveContainer" containerID="ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e" Oct 30 04:35:03.636478 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:35:03.636458 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e\": container with ID starting with ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e not found: ID does not exist" containerID="ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e" Oct 30 04:35:03.636529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.636486 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e"} err="failed to get container status \"ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e\": rpc error: code = NotFound desc = could not find container \"ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e\": container with ID starting with ae3944dc00ef93585dbf9fa3ee46a51fb862c8fe3ac735a28614c0ec168a624e not found: ID does not exist" Oct 30 04:35:03.636529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.636504 4793 scope.go:122] "RemoveContainer" containerID="1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63" Oct 30 04:35:03.636748 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:35:03.636730 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63\": container with ID starting with 1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63 not found: ID does not exist" containerID="1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63" Oct 30 04:35:03.636789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.636755 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63"} err="failed to get container status \"1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63\": rpc error: code = NotFound desc = could not find container \"1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63\": container with ID starting with 1e9e5bf8b6f774030d07cecb03b7d8e4b93a7ed01e8881ce9368c238dc818b63 not found: ID does not exist" Oct 30 04:35:03.687095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.687066 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x"] Oct 30 04:35:03.699007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:03.698979 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-86eaa9-openshift-default-558749b8bf-nq55x"] Oct 30 04:35:04.045929 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:04.045879 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:04.596659 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:04.596620 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h"] Oct 30 04:35:04.629960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:04.629927 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6"] Oct 30 04:35:04.846580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:04.846548 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" path="/var/lib/kubelet/pods/ac8095c1-91a7-4082-84d2-e52b612e736b/volumes" Oct 30 04:35:04.950458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:04.950398 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:05.163319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:05.163288 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:05.915170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:05.915124 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:07.703949 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:07.703897 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:09.006144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.006103 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn"] Oct 30 04:35:09.006620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.006367 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" podUID="e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2" containerName="maas-api" containerID="cri-o://6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0" gracePeriod=30 Oct 30 04:35:09.172400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.172367 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:09.340585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.340559 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:35:09.503194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.503165 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-maas-api-tls\") pod \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " Oct 30 04:35:09.503494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.503241 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-openshift-service-ca\") pod \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " Oct 30 04:35:09.503494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.503293 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-kube-api-access-m5lfl\" (UniqueName: \"kubernetes.io/projected/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-kube-api-access-m5lfl\") pod \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " Oct 30 04:35:09.503494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.503397 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-metrics-certs\") pod \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\" (UID: \"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2\") " Oct 30 04:35:09.503783 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.503752 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-openshift-service-ca" pod "e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2" (UID: "e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:35:09.504097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.504070 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:09.505807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.505782 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-kube-api-access-m5lfl" pod "e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2" (UID: "e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2"). InnerVolumeSpecName "kube-api-access-m5lfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:35:09.507863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.507835 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-maas-api-tls" pod "e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2" (UID: "e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:35:09.507967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.507891 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-metrics-certs" pod "e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2" (UID: "e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:35:09.597197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.597128 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" podUID="267efa5e-64d0-41a1-a43e-28b81ed93bd9" containerName="payload-processing" containerID="cri-o://b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079" gracePeriod=25 Oct 30 04:35:09.604817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.604794 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5lfl\" (UniqueName: \"kubernetes.io/projected/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-kube-api-access-m5lfl\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:09.604898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.604819 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:09.604898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.604834 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:09.631025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.630988 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" podUID="c8223981-1aa9-439f-bd78-f7b2edeaaa29" containerName="payload-pre-processing" containerID="cri-o://ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710" gracePeriod=25 Oct 30 04:35:09.663552 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.663521 4793 generic.go:356] "Generic (PLEG): container finished" podID="e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2" containerID="6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0" exitCode=0 Oct 30 04:35:09.663677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.663591 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" Oct 30 04:35:09.663677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.663653 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" event={"ID":"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2","Type":"ContainerDied","Data":"6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0"} Oct 30 04:35:09.663808 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.663709 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn" event={"ID":"e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2","Type":"ContainerDied","Data":"592a528bea9b546c19bbe667580c9fc2c8fd6dfe94250a4ab3fb0199bfa83e1e"} Oct 30 04:35:09.663808 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.663737 4793 scope.go:122] "RemoveContainer" containerID="6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0" Oct 30 04:35:09.743617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.743590 4793 scope.go:122] "RemoveContainer" containerID="6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0" Oct 30 04:35:09.743961 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:35:09.743937 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0\": container with ID starting with 6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0 not found: ID does not exist" containerID="6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0" Oct 30 04:35:09.744049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.743971 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0"} err="failed to get container status \"6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0\": rpc error: code = NotFound desc = could not find container \"6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0\": container with ID starting with 6a7e6447ff23266f0840117e052c4190d3dd63b9683acbb020038e4c7e51afa0 not found: ID does not exist" Oct 30 04:35:09.757840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.757812 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn"] Oct 30 04:35:09.784999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.784957 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-2696d8-b6d98b57b-r44bn"] Oct 30 04:35:09.842471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.838823 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:09.965943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:09.962341 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" Oct 30 04:35:10.013984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.013950 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/267efa5e-64d0-41a1-a43e-28b81ed93bd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267efa5e-64d0-41a1-a43e-28b81ed93bd9-plugins-config-volume\") pod \"267efa5e-64d0-41a1-a43e-28b81ed93bd9\" (UID: \"267efa5e-64d0-41a1-a43e-28b81ed93bd9\") " Oct 30 04:35:10.014473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.014357 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/267efa5e-64d0-41a1-a43e-28b81ed93bd9-plugins-config-volume" pod "267efa5e-64d0-41a1-a43e-28b81ed93bd9" (UID: "267efa5e-64d0-41a1-a43e-28b81ed93bd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:35:10.014639 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.014613 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/267efa5e-64d0-41a1-a43e-28b81ed93bd9-kube-api-access-qqwrw\" (UniqueName: \"kubernetes.io/projected/267efa5e-64d0-41a1-a43e-28b81ed93bd9-kube-api-access-qqwrw\") pod \"267efa5e-64d0-41a1-a43e-28b81ed93bd9\" (UID: \"267efa5e-64d0-41a1-a43e-28b81ed93bd9\") " Oct 30 04:35:10.015130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.015106 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/267efa5e-64d0-41a1-a43e-28b81ed93bd9-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:10.017342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.017320 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/267efa5e-64d0-41a1-a43e-28b81ed93bd9-kube-api-access-qqwrw" pod "267efa5e-64d0-41a1-a43e-28b81ed93bd9" (UID: "267efa5e-64d0-41a1-a43e-28b81ed93bd9"). InnerVolumeSpecName "kube-api-access-qqwrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:35:10.055788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.055768 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" Oct 30 04:35:10.124373 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.124339 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqwrw\" (UniqueName: \"kubernetes.io/projected/267efa5e-64d0-41a1-a43e-28b81ed93bd9-kube-api-access-qqwrw\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:10.225317 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.225286 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8223981-1aa9-439f-bd78-f7b2edeaaa29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8223981-1aa9-439f-bd78-f7b2edeaaa29-plugins-config-volume\") pod \"c8223981-1aa9-439f-bd78-f7b2edeaaa29\" (UID: \"c8223981-1aa9-439f-bd78-f7b2edeaaa29\") " Oct 30 04:35:10.225527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.225501 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8223981-1aa9-439f-bd78-f7b2edeaaa29-kube-api-access-2z868\" (UniqueName: \"kubernetes.io/projected/c8223981-1aa9-439f-bd78-f7b2edeaaa29-kube-api-access-2z868\") pod \"c8223981-1aa9-439f-bd78-f7b2edeaaa29\" (UID: \"c8223981-1aa9-439f-bd78-f7b2edeaaa29\") " Oct 30 04:35:10.225696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.225666 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8223981-1aa9-439f-bd78-f7b2edeaaa29-plugins-config-volume" pod "c8223981-1aa9-439f-bd78-f7b2edeaaa29" (UID: "c8223981-1aa9-439f-bd78-f7b2edeaaa29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:35:10.225893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.225869 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8223981-1aa9-439f-bd78-f7b2edeaaa29-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:10.227723 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.227701 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8223981-1aa9-439f-bd78-f7b2edeaaa29-kube-api-access-2z868" pod "c8223981-1aa9-439f-bd78-f7b2edeaaa29" (UID: "c8223981-1aa9-439f-bd78-f7b2edeaaa29"). InnerVolumeSpecName "kube-api-access-2z868". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:35:10.326971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.326943 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z868\" (UniqueName: \"kubernetes.io/projected/c8223981-1aa9-439f-bd78-f7b2edeaaa29-kube-api-access-2z868\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:10.671838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.671746 4793 generic.go:356] "Generic (PLEG): container finished" podID="267efa5e-64d0-41a1-a43e-28b81ed93bd9" containerID="b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079" exitCode=0 Oct 30 04:35:10.671838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.671789 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" Oct 30 04:35:10.671838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.671826 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" event={"ID":"267efa5e-64d0-41a1-a43e-28b81ed93bd9","Type":"ContainerDied","Data":"b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079"} Oct 30 04:35:10.672120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.671907 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h" event={"ID":"267efa5e-64d0-41a1-a43e-28b81ed93bd9","Type":"ContainerDied","Data":"ff02922dbfc26ee4c38791b6741a0dc925017610273385fc08eda3795e587e8d"} Oct 30 04:35:10.672120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.671931 4793 scope.go:122] "RemoveContainer" containerID="b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079" Oct 30 04:35:10.674958 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.674932 4793 generic.go:356] "Generic (PLEG): container finished" podID="c8223981-1aa9-439f-bd78-f7b2edeaaa29" containerID="ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710" exitCode=0 Oct 30 04:35:10.675077 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.674962 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" event={"ID":"c8223981-1aa9-439f-bd78-f7b2edeaaa29","Type":"ContainerDied","Data":"ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710"} Oct 30 04:35:10.675077 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.674987 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" Oct 30 04:35:10.675077 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.675009 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6" event={"ID":"c8223981-1aa9-439f-bd78-f7b2edeaaa29","Type":"ContainerDied","Data":"e34243aa06af527fe9b896ad8bc1d5cdfb603a2e2edf290da81899292975bfd8"} Oct 30 04:35:10.682686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.682664 4793 scope.go:122] "RemoveContainer" containerID="b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079" Oct 30 04:35:10.682954 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:35:10.682930 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079\": container with ID starting with b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079 not found: ID does not exist" containerID="b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079" Oct 30 04:35:10.683021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.682963 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079"} err="failed to get container status \"b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079\": rpc error: code = NotFound desc = could not find container \"b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079\": container with ID starting with b5ff02904a810dd04fa679c531f4d7b2acdbaac70f8f58d15aa5514843863079 not found: ID does not exist" Oct 30 04:35:10.683021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.682980 4793 scope.go:122] "RemoveContainer" containerID="ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710" Oct 30 04:35:10.692039 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.692016 4793 scope.go:122] "RemoveContainer" containerID="ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710" Oct 30 04:35:10.692291 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:35:10.692272 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710\": container with ID starting with ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710 not found: ID does not exist" containerID="ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710" Oct 30 04:35:10.692351 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.692297 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710"} err="failed to get container status \"ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710\": rpc error: code = NotFound desc = could not find container \"ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710\": container with ID starting with ec24d3e11e80884eec0ee2d8c560025bc5cb44c7624f8e596aca1b3e386ac710 not found: ID does not exist" Oct 30 04:35:10.737567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.737542 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6"] Oct 30 04:35:10.749402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.749378 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-2696d8-5f8657f5bc-tssf6"] Oct 30 04:35:10.780758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.780729 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h"] Oct 30 04:35:10.785863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.785840 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-2696d8-6b6c64c76-hfq5h"] Oct 30 04:35:10.845932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.845898 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="267efa5e-64d0-41a1-a43e-28b81ed93bd9" path="/var/lib/kubelet/pods/267efa5e-64d0-41a1-a43e-28b81ed93bd9/volumes" Oct 30 04:35:10.846346 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.846325 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8223981-1aa9-439f-bd78-f7b2edeaaa29" path="/var/lib/kubelet/pods/c8223981-1aa9-439f-bd78-f7b2edeaaa29/volumes" Oct 30 04:35:10.846795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:10.846775 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2" path="/var/lib/kubelet/pods/e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2/volumes" Oct 30 04:35:11.225213 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:35:11.225175 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:35:11.252486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:11.252454 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:11.470461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:11.469531 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:11.689025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:11.688990 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:11.963652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:11.963564 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:12.225404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:12.225322 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt"] Oct 30 04:35:12.225893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:12.225591 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" containerID="cri-o://ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc" gracePeriod=30 Oct 30 04:35:12.401477 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:12.400452 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:13.419317 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:13.419281 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:15.111240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:15.111200 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:16.792164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:16.792094 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:17.511301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.511246 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:35:17.621197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.621159 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-token\" (UniqueName: \"kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-token\") pod \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " Oct 30 04:35:17.621361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.621204 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-socket\") pod \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " Oct 30 04:35:17.621361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.621234 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-credential-socket\") pod \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " Oct 30 04:35:17.621361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.621337 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-podinfo\") pod \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " Oct 30 04:35:17.621570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.621380 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-certs\") pod \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " Oct 30 04:35:17.621570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.621409 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istiod-ca-cert\") pod \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " Oct 30 04:35:17.621692 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.621585 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-socket" pod "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" (UID: "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:17.621692 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.621603 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-credential-socket" pod "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" (UID: "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:17.621803 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.621734 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-certs" pod "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" (UID: "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:17.621844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.621823 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istiod-ca-cert" pod "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" (UID: "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:35:17.621926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.621913 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-kube-api-access-v54l2\" (UniqueName: \"kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-kube-api-access-v54l2\") pod \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " Oct 30 04:35:17.621974 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.621949 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-data\") pod \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " Oct 30 04:35:17.622033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.622022 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-envoy\") pod \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\" (UID: \"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e\") " Oct 30 04:35:17.622247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.622218 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-data" pod "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" (UID: "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:17.622685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.622665 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:17.622840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.622818 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:17.622840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.622839 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:17.623041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.622852 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:17.623041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.622865 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:17.623665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.623635 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-podinfo" pod "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" (UID: "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:35:17.623766 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.623736 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-token" pod "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" (UID: "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:35:17.624396 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.624369 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-kube-api-access-v54l2" pod "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" (UID: "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e"). InnerVolumeSpecName "kube-api-access-v54l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:35:17.624498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.624452 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-envoy" pod "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" (UID: "ecf9878b-08a0-4f63-8e02-f6b8c25ef88e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:17.724213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.724176 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:17.724213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.724205 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:17.724213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.724214 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:17.724530 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.724227 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v54l2\" (UniqueName: \"kubernetes.io/projected/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e-kube-api-access-v54l2\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:17.726008 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.725979 4793 generic.go:356] "Generic (PLEG): container finished" podID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerID="ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc" exitCode=0 Oct 30 04:35:17.726120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.726063 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" Oct 30 04:35:17.726120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.726081 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" event={"ID":"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e","Type":"ContainerDied","Data":"ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc"} Oct 30 04:35:17.726188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.726134 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt" event={"ID":"ecf9878b-08a0-4f63-8e02-f6b8c25ef88e","Type":"ContainerDied","Data":"5b0d7a7a7f29177c72dbf6c6819304884502e4c62b236d493a61ed0f5fd7a6b7"} Oct 30 04:35:17.726188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.726160 4793 scope.go:122] "RemoveContainer" containerID="ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc" Oct 30 04:35:17.737594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.737578 4793 scope.go:122] "RemoveContainer" containerID="323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e" Oct 30 04:35:17.747330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.747312 4793 scope.go:122] "RemoveContainer" containerID="ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc" Oct 30 04:35:17.747686 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:35:17.747664 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc\": container with ID starting with ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc not found: ID does not exist" containerID="ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc" Oct 30 04:35:17.747754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.747695 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc"} err="failed to get container status \"ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc\": rpc error: code = NotFound desc = could not find container \"ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc\": container with ID starting with ca82a6ab7370570fa18bb1746efe56ef06cb0cd4cfebf77e8749c9200513f2fc not found: ID does not exist" Oct 30 04:35:17.747754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.747716 4793 scope.go:122] "RemoveContainer" containerID="323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e" Oct 30 04:35:17.747962 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:35:17.747942 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e\": container with ID starting with 323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e not found: ID does not exist" containerID="323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e" Oct 30 04:35:17.748004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.747970 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e"} err="failed to get container status \"323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e\": rpc error: code = NotFound desc = could not find container \"323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e\": container with ID starting with 323337b0f4ccef0a5e9d9a04f4c694a92c51d41850eac51c66fae47ca695758e not found: ID does not exist" Oct 30 04:35:17.784934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.784852 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt"] Oct 30 04:35:17.788939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:17.788913 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-2696d8-openshift-default-6c5d78dd6f-t64mt"] Oct 30 04:35:18.586767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:18.586727 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:18.847172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:18.847094 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" path="/var/lib/kubelet/pods/ecf9878b-08a0-4f63-8e02-f6b8c25ef88e/volumes" Oct 30 04:35:19.158262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158176 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl"] Oct 30 04:35:19.158754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158737 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158755 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158769 4793 cpu_manager.go:414] "containerMap: removing container" podUID="eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7" containerName="payload-pre-processing" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158774 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7" containerName="payload-pre-processing" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158791 4793 cpu_manager.go:414] "containerMap: removing container" podUID="53227857-f430-41a9-9a08-3e2daad9f1b7" containerName="payload-processing" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158798 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53227857-f430-41a9-9a08-3e2daad9f1b7" containerName="payload-processing" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158818 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerName="istio-proxy" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158823 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerName="istio-proxy" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158829 4793 cpu_manager.go:414] "containerMap: removing container" podUID="73e08ad2-cda0-4914-b839-081b2f791a82" containerName="payload-processing" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158834 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e08ad2-cda0-4914-b839-081b2f791a82" containerName="payload-processing" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158839 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerName="istio-proxy" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158843 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerName="istio-proxy" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158849 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c8223981-1aa9-439f-bd78-f7b2edeaaa29" containerName="payload-pre-processing" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158853 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8223981-1aa9-439f-bd78-f7b2edeaaa29" containerName="payload-pre-processing" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158869 4793 cpu_manager.go:414] "containerMap: removing container" podUID="83183572-0667-4816-88b4-4f85b8aa041a" containerName="revoke-keys" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158878 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83183572-0667-4816-88b4-4f85b8aa041a" containerName="revoke-keys" Oct 30 04:35:19.158881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158886 4793 cpu_manager.go:414] "containerMap: removing container" podUID="267efa5e-64d0-41a1-a43e-28b81ed93bd9" containerName="payload-processing" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158891 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="267efa5e-64d0-41a1-a43e-28b81ed93bd9" containerName="payload-processing" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158902 4793 cpu_manager.go:414] "containerMap: removing container" podUID="e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2" containerName="maas-api" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158916 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2" containerName="maas-api" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.158998 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="53227857-f430-41a9-9a08-3e2daad9f1b7" containerName="payload-processing" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.159009 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerName="istio-proxy" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.159014 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e08ad2-cda0-4914-b839-081b2f791a82" containerName="payload-processing" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.159019 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8095c1-91a7-4082-84d2-e52b612e736b" containerName="istio-proxy" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.159027 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8223981-1aa9-439f-bd78-f7b2edeaaa29" containerName="payload-pre-processing" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.159036 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.159041 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="83183572-0667-4816-88b4-4f85b8aa041a" containerName="revoke-keys" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.159047 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="267efa5e-64d0-41a1-a43e-28b81ed93bd9" containerName="payload-processing" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.159054 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1e1dbe7-9bd2-49c3-9ba9-6db2fdf6e7e2" containerName="maas-api" Oct 30 04:35:19.159677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.159065 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafd8e6b-2d91-40b5-84fc-e4ddc3b029d7" containerName="payload-pre-processing" Oct 30 04:35:19.164041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.164020 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:19.181654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.181623 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl"] Oct 30 04:35:19.192453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.189672 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:35:19.192453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.189764 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:35:19.192453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.189893 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:35:19.192453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.189949 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:35:19.240612 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.240578 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ba18b8d-dgjsl\" (UID: \"0a28ba79-5e26-4973-b99d-5b97dc185d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:19.240756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.240729 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52v4l\" (UniqueName: \"kubernetes.io/projected/0a28ba79-5e26-4973-b99d-5b97dc185d61-kube-api-access-52v4l\") pod \"maas-api-revoke-sub-5ba18b8d-dgjsl\" (UID: \"0a28ba79-5e26-4973-b99d-5b97dc185d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:19.240825 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.240808 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ba18b8d-dgjsl\" (UID: \"0a28ba79-5e26-4973-b99d-5b97dc185d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:19.341984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.341949 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ba18b8d-dgjsl\" (UID: \"0a28ba79-5e26-4973-b99d-5b97dc185d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:19.342146 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.342045 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ba18b8d-dgjsl\" (UID: \"0a28ba79-5e26-4973-b99d-5b97dc185d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:19.342190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.342171 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52v4l\" (UniqueName: \"kubernetes.io/projected/0a28ba79-5e26-4973-b99d-5b97dc185d61-kube-api-access-52v4l\") pod \"maas-api-revoke-sub-5ba18b8d-dgjsl\" (UID: \"0a28ba79-5e26-4973-b99d-5b97dc185d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:19.342747 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.342722 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ba18b8d-dgjsl\" (UID: \"0a28ba79-5e26-4973-b99d-5b97dc185d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:19.344259 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.344235 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ba18b8d-dgjsl\" (UID: \"0a28ba79-5e26-4973-b99d-5b97dc185d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:19.368068 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.368039 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52v4l\" (UniqueName: \"kubernetes.io/projected/0a28ba79-5e26-4973-b99d-5b97dc185d61-kube-api-access-52v4l\") pod \"maas-api-revoke-sub-5ba18b8d-dgjsl\" (UID: \"0a28ba79-5e26-4973-b99d-5b97dc185d61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:19.477980 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.477950 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:19.781718 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:19.781685 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl"] Oct 30 04:35:19.782301 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:35:19.782279 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a28ba79_5e26_4973_b99d_5b97dc185d61.slice/crio-7c95edc7c56a810daf8f70ee1b1d4a8030f2f526aaf4a2f49662111a363236bf WatchSource:0}: Error finding container 7c95edc7c56a810daf8f70ee1b1d4a8030f2f526aaf4a2f49662111a363236bf: Status 404 returned error can't find the container with id 7c95edc7c56a810daf8f70ee1b1d4a8030f2f526aaf4a2f49662111a363236bf Oct 30 04:35:20.377855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:20.377772 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:20.752251 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:20.752217 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" event={"ID":"0a28ba79-5e26-4973-b99d-5b97dc185d61","Type":"ContainerStarted","Data":"90e6d62cef63ac368ba7b4b091007914490ae85e9fc8f9ef2d939fe1197f8908"} Oct 30 04:35:20.752450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:20.752276 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" event={"ID":"0a28ba79-5e26-4973-b99d-5b97dc185d61","Type":"ContainerStarted","Data":"7c95edc7c56a810daf8f70ee1b1d4a8030f2f526aaf4a2f49662111a363236bf"} Oct 30 04:35:20.793795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:20.793750 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" podStartSLOduration=1.7937375389999999 podStartE2EDuration="1.793737539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:35:20.79281499 +0000 UTC m=+2244.078689764" watchObservedRunningTime="2026-10-30 04:35:20.793737539 +0000 UTC m=+2244.079612313" Oct 30 04:35:21.759836 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:21.759801 4793 generic.go:356] "Generic (PLEG): container finished" podID="0a28ba79-5e26-4973-b99d-5b97dc185d61" containerID="90e6d62cef63ac368ba7b4b091007914490ae85e9fc8f9ef2d939fe1197f8908" exitCode=0 Oct 30 04:35:21.760028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:21.759882 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" event={"ID":"0a28ba79-5e26-4973-b99d-5b97dc185d61","Type":"ContainerDied","Data":"90e6d62cef63ac368ba7b4b091007914490ae85e9fc8f9ef2d939fe1197f8908"} Oct 30 04:35:21.940469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:21.940428 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:22.084772 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:22.084695 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk"] Oct 30 04:35:22.085009 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:22.084929 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" podUID="70316b40-a67f-477f-886a-96779807d046" containerName="istio-proxy" containerID="cri-o://282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca" gracePeriod=30 Oct 30 04:35:22.919859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:22.919834 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:23.012914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.012887 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-xks-ca\") pod \"0a28ba79-5e26-4973-b99d-5b97dc185d61\" (UID: \"0a28ba79-5e26-4973-b99d-5b97dc185d61\") " Oct 30 04:35:23.013281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.013052 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-openshift-ca\") pod \"0a28ba79-5e26-4973-b99d-5b97dc185d61\" (UID: \"0a28ba79-5e26-4973-b99d-5b97dc185d61\") " Oct 30 04:35:23.013281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.013084 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a28ba79-5e26-4973-b99d-5b97dc185d61-kube-api-access-52v4l\" (UniqueName: \"kubernetes.io/projected/0a28ba79-5e26-4973-b99d-5b97dc185d61-kube-api-access-52v4l\") pod \"0a28ba79-5e26-4973-b99d-5b97dc185d61\" (UID: \"0a28ba79-5e26-4973-b99d-5b97dc185d61\") " Oct 30 04:35:23.013497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.013440 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-openshift-ca" pod "0a28ba79-5e26-4973-b99d-5b97dc185d61" (UID: "0a28ba79-5e26-4973-b99d-5b97dc185d61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:35:23.013751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.013728 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:23.015174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.015146 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-xks-ca" pod "0a28ba79-5e26-4973-b99d-5b97dc185d61" (UID: "0a28ba79-5e26-4973-b99d-5b97dc185d61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:35:23.015274 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.015207 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a28ba79-5e26-4973-b99d-5b97dc185d61-kube-api-access-52v4l" pod "0a28ba79-5e26-4973-b99d-5b97dc185d61" (UID: "0a28ba79-5e26-4973-b99d-5b97dc185d61"). InnerVolumeSpecName "kube-api-access-52v4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:35:23.114636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.114603 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a28ba79-5e26-4973-b99d-5b97dc185d61-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:23.114636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.114634 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52v4l\" (UniqueName: \"kubernetes.io/projected/0a28ba79-5e26-4973-b99d-5b97dc185d61-kube-api-access-52v4l\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:23.613575 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.613539 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:23.775035 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.775001 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" event={"ID":"0a28ba79-5e26-4973-b99d-5b97dc185d61","Type":"ContainerDied","Data":"7c95edc7c56a810daf8f70ee1b1d4a8030f2f526aaf4a2f49662111a363236bf"} Oct 30 04:35:23.775183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.775045 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl" Oct 30 04:35:23.775183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:23.775049 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c95edc7c56a810daf8f70ee1b1d4a8030f2f526aaf4a2f49662111a363236bf" Oct 30 04:35:25.307594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:25.307554 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:26.921781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:26.921746 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:27.371202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.371177 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:35:27.465807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.465714 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-certs\") pod \"70316b40-a67f-477f-886a-96779807d046\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " Oct 30 04:35:27.465807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.465792 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-istio-token\" (UniqueName: \"kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-istio-token\") pod \"70316b40-a67f-477f-886a-96779807d046\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " Oct 30 04:35:27.466048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.465831 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-socket\") pod \"70316b40-a67f-477f-886a-96779807d046\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " Oct 30 04:35:27.466048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.465879 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-data\") pod \"70316b40-a67f-477f-886a-96779807d046\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " Oct 30 04:35:27.466048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.465919 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-kube-api-access-jm4mg\" (UniqueName: \"kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-kube-api-access-jm4mg\") pod \"70316b40-a67f-477f-886a-96779807d046\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " Oct 30 04:35:27.466048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.465964 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70316b40-a67f-477f-886a-96779807d046-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70316b40-a67f-477f-886a-96779807d046-istio-podinfo\") pod \"70316b40-a67f-477f-886a-96779807d046\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " Oct 30 04:35:27.466048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.465998 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-credential-socket\") pod \"70316b40-a67f-477f-886a-96779807d046\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " Oct 30 04:35:27.466048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.466015 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-certs" pod "70316b40-a67f-477f-886a-96779807d046" (UID: "70316b40-a67f-477f-886a-96779807d046"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:27.466363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.466108 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-envoy\") pod \"70316b40-a67f-477f-886a-96779807d046\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " Oct 30 04:35:27.466363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.466192 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70316b40-a67f-477f-886a-96779807d046-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70316b40-a67f-477f-886a-96779807d046-istiod-ca-cert\") pod \"70316b40-a67f-477f-886a-96779807d046\" (UID: \"70316b40-a67f-477f-886a-96779807d046\") " Oct 30 04:35:27.466363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.466300 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-credential-socket" pod "70316b40-a67f-477f-886a-96779807d046" (UID: "70316b40-a67f-477f-886a-96779807d046"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:27.466920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.466711 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:27.466920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.466738 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:27.466920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.466860 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-data" pod "70316b40-a67f-477f-886a-96779807d046" (UID: "70316b40-a67f-477f-886a-96779807d046"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:27.467166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.467056 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-socket" pod "70316b40-a67f-477f-886a-96779807d046" (UID: "70316b40-a67f-477f-886a-96779807d046"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:27.467677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.467382 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70316b40-a67f-477f-886a-96779807d046-istiod-ca-cert" pod "70316b40-a67f-477f-886a-96779807d046" (UID: "70316b40-a67f-477f-886a-96779807d046"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:35:27.469468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.469400 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-kube-api-access-jm4mg" pod "70316b40-a67f-477f-886a-96779807d046" (UID: "70316b40-a67f-477f-886a-96779807d046"). InnerVolumeSpecName "kube-api-access-jm4mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:35:27.469724 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.469678 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-istio-token" pod "70316b40-a67f-477f-886a-96779807d046" (UID: "70316b40-a67f-477f-886a-96779807d046"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:35:27.470169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.470144 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-envoy" pod "70316b40-a67f-477f-886a-96779807d046" (UID: "70316b40-a67f-477f-886a-96779807d046"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:35:27.471794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.471648 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70316b40-a67f-477f-886a-96779807d046-istio-podinfo" pod "70316b40-a67f-477f-886a-96779807d046" (UID: "70316b40-a67f-477f-886a-96779807d046"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:35:27.567726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.567698 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:27.567726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.567724 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm4mg\" (UniqueName: \"kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-kube-api-access-jm4mg\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:27.567924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.567735 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70316b40-a67f-477f-886a-96779807d046-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:27.567924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.567744 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:27.567924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.567753 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70316b40-a67f-477f-886a-96779807d046-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:27.567924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.567760 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70316b40-a67f-477f-886a-96779807d046-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:27.567924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.567767 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70316b40-a67f-477f-886a-96779807d046-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:27.803169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.803080 4793 generic.go:356] "Generic (PLEG): container finished" podID="70316b40-a67f-477f-886a-96779807d046" containerID="282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca" exitCode=0 Oct 30 04:35:27.803169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.803124 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" Oct 30 04:35:27.803169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.803141 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" event={"ID":"70316b40-a67f-477f-886a-96779807d046","Type":"ContainerDied","Data":"282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca"} Oct 30 04:35:27.803446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.803195 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk" event={"ID":"70316b40-a67f-477f-886a-96779807d046","Type":"ContainerDied","Data":"96da11553e15f1f4f7d256a03ede9525743f42e95ecc9525f7ca0e9e50d3742a"} Oct 30 04:35:27.803446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.803223 4793 scope.go:122] "RemoveContainer" containerID="282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca" Oct 30 04:35:27.813393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.813372 4793 scope.go:122] "RemoveContainer" containerID="282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca" Oct 30 04:35:27.813714 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:35:27.813696 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca\": container with ID starting with 282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca not found: ID does not exist" containerID="282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca" Oct 30 04:35:27.813789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.813724 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca"} err="failed to get container status \"282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca\": rpc error: code = NotFound desc = could not find container \"282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca\": container with ID starting with 282d4a3781fb249a77436b86e8d526620960b66b437214d497c92743489203ca not found: ID does not exist" Oct 30 04:35:27.859198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.859172 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk"] Oct 30 04:35:27.869861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:27.869834 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78gqk"] Oct 30 04:35:28.607040 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:28.606944 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:28.845935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:28.845903 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70316b40-a67f-477f-886a-96779807d046" path="/var/lib/kubelet/pods/70316b40-a67f-477f-886a-96779807d046/volumes" Oct 30 04:35:30.341484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:30.341451 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:32.039537 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:32.039486 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:33.340817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.340781 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9"] Oct 30 04:35:33.341293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.341278 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0a28ba79-5e26-4973-b99d-5b97dc185d61" containerName="revoke-keys" Oct 30 04:35:33.341337 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.341294 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a28ba79-5e26-4973-b99d-5b97dc185d61" containerName="revoke-keys" Oct 30 04:35:33.341337 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.341313 4793 cpu_manager.go:414] "containerMap: removing container" podUID="70316b40-a67f-477f-886a-96779807d046" containerName="istio-proxy" Oct 30 04:35:33.341337 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.341318 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70316b40-a67f-477f-886a-96779807d046" containerName="istio-proxy" Oct 30 04:35:33.341337 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.341334 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" Oct 30 04:35:33.341484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.341340 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" Oct 30 04:35:33.341484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.341432 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a28ba79-5e26-4973-b99d-5b97dc185d61" containerName="revoke-keys" Oct 30 04:35:33.341484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.341440 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf9878b-08a0-4f63-8e02-f6b8c25ef88e" containerName="istio-proxy" Oct 30 04:35:33.341484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.341449 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="70316b40-a67f-477f-886a-96779807d046" containerName="istio-proxy" Oct 30 04:35:33.345930 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.345910 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:33.363709 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.363683 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9"] Oct 30 04:35:33.371053 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.371027 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:35:33.371230 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.371116 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:35:33.371902 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.371873 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:35:33.373528 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.373507 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:35:33.439170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.439135 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e939b9a-4h8g9\" (UID: \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:33.439325 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.439188 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e939b9a-4h8g9\" (UID: \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:33.439438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.439368 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7c57\" (UniqueName: \"kubernetes.io/projected/09401b03-488c-4420-9f0a-d8e6c7ce6aef-kube-api-access-s7c57\") pod \"maas-api-revoke-sub-5e939b9a-4h8g9\" (UID: \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:33.540975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.540938 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e939b9a-4h8g9\" (UID: \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:33.541150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.540991 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e939b9a-4h8g9\" (UID: \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:33.541150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.541067 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7c57\" (UniqueName: \"kubernetes.io/projected/09401b03-488c-4420-9f0a-d8e6c7ce6aef-kube-api-access-s7c57\") pod \"maas-api-revoke-sub-5e939b9a-4h8g9\" (UID: \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:33.541868 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.541846 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e939b9a-4h8g9\" (UID: \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:33.543396 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.543359 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e939b9a-4h8g9\" (UID: \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:33.564343 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.564322 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7c57\" (UniqueName: \"kubernetes.io/projected/09401b03-488c-4420-9f0a-d8e6c7ce6aef-kube-api-access-s7c57\") pod \"maas-api-revoke-sub-5e939b9a-4h8g9\" (UID: \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:33.659358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.659302 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:33.825693 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.825647 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:33.961980 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:33.961950 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9"] Oct 30 04:35:33.962609 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:35:33.962587 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09401b03_488c_4420_9f0a_d8e6c7ce6aef.slice/crio-b9723320a0949bd7876b4eb05fea8369f62bfeef54bdeed2932e128b594e777a WatchSource:0}: Error finding container b9723320a0949bd7876b4eb05fea8369f62bfeef54bdeed2932e128b594e777a: Status 404 returned error can't find the container with id b9723320a0949bd7876b4eb05fea8369f62bfeef54bdeed2932e128b594e777a Oct 30 04:35:34.851496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:34.851458 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" event={"ID":"09401b03-488c-4420-9f0a-d8e6c7ce6aef","Type":"ContainerStarted","Data":"2c4c29ec4dba66a9213281e83d7d23b79555d56deb932495fb2e1647604ec05b"} Oct 30 04:35:34.851985 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:34.851519 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" event={"ID":"09401b03-488c-4420-9f0a-d8e6c7ce6aef","Type":"ContainerStarted","Data":"b9723320a0949bd7876b4eb05fea8369f62bfeef54bdeed2932e128b594e777a"} Oct 30 04:35:35.516664 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:35.516619 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:35.860705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:35.860616 4793 generic.go:356] "Generic (PLEG): container finished" podID="09401b03-488c-4420-9f0a-d8e6c7ce6aef" containerID="2c4c29ec4dba66a9213281e83d7d23b79555d56deb932495fb2e1647604ec05b" exitCode=0 Oct 30 04:35:35.860705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:35.860675 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" event={"ID":"09401b03-488c-4420-9f0a-d8e6c7ce6aef","Type":"ContainerDied","Data":"2c4c29ec4dba66a9213281e83d7d23b79555d56deb932495fb2e1647604ec05b"} Oct 30 04:35:37.019115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.019087 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:37.087734 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.087693 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09401b03-488c-4420-9f0a-d8e6c7ce6aef-kube-api-access-s7c57\" (UniqueName: \"kubernetes.io/projected/09401b03-488c-4420-9f0a-d8e6c7ce6aef-kube-api-access-s7c57\") pod \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\" (UID: \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\") " Oct 30 04:35:37.088897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.087820 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-openshift-ca\") pod \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\" (UID: \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\") " Oct 30 04:35:37.088897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.087878 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-xks-ca\") pod \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\" (UID: \"09401b03-488c-4420-9f0a-d8e6c7ce6aef\") " Oct 30 04:35:37.099691 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.089754 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-openshift-ca" pod "09401b03-488c-4420-9f0a-d8e6c7ce6aef" (UID: "09401b03-488c-4420-9f0a-d8e6c7ce6aef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:35:37.099691 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.094789 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-xks-ca" pod "09401b03-488c-4420-9f0a-d8e6c7ce6aef" (UID: "09401b03-488c-4420-9f0a-d8e6c7ce6aef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:35:37.099691 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.097932 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09401b03-488c-4420-9f0a-d8e6c7ce6aef-kube-api-access-s7c57" pod "09401b03-488c-4420-9f0a-d8e6c7ce6aef" (UID: "09401b03-488c-4420-9f0a-d8e6c7ce6aef"). InnerVolumeSpecName "kube-api-access-s7c57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:35:37.118982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.118908 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:37.190184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.190153 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7c57\" (UniqueName: \"kubernetes.io/projected/09401b03-488c-4420-9f0a-d8e6c7ce6aef-kube-api-access-s7c57\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:37.190184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.190186 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:37.190353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.190202 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09401b03-488c-4420-9f0a-d8e6c7ce6aef-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:37.875798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.875765 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" Oct 30 04:35:37.875798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.875786 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9" event={"ID":"09401b03-488c-4420-9f0a-d8e6c7ce6aef","Type":"ContainerDied","Data":"b9723320a0949bd7876b4eb05fea8369f62bfeef54bdeed2932e128b594e777a"} Oct 30 04:35:37.876026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:37.875828 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9723320a0949bd7876b4eb05fea8369f62bfeef54bdeed2932e128b594e777a" Oct 30 04:35:38.829010 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:38.828975 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:40.524484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:40.524442 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:41.218685 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:35:41.218649 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:35:42.250395 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:42.250360 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:42.992704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:42.992670 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:43.220470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:43.220433 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:43.913763 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:43.913727 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:45.661179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:45.661139 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:47.313191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.313152 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:47.469739 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.469707 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr"] Oct 30 04:35:47.470506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.470465 4793 cpu_manager.go:414] "containerMap: removing container" podUID="09401b03-488c-4420-9f0a-d8e6c7ce6aef" containerName="revoke-keys" Oct 30 04:35:47.470506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.470485 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09401b03-488c-4420-9f0a-d8e6c7ce6aef" containerName="revoke-keys" Oct 30 04:35:47.470705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.470606 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="09401b03-488c-4420-9f0a-d8e6c7ce6aef" containerName="revoke-keys" Oct 30 04:35:47.474111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.474092 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:47.521261 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.521220 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr"] Oct 30 04:35:47.533458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.532479 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:35:47.533458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.532482 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:35:47.533458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.532557 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:35:47.534218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.534195 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:35:47.607676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.607589 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e68ae2fe-2zxxr\" (UID: \"46944f21-f6bf-41fc-af89-392daeb9693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:47.607833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.607730 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsldl\" (UniqueName: \"kubernetes.io/projected/46944f21-f6bf-41fc-af89-392daeb9693f-kube-api-access-nsldl\") pod \"maas-api-revoke-sub-e68ae2fe-2zxxr\" (UID: \"46944f21-f6bf-41fc-af89-392daeb9693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:47.607833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.607765 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e68ae2fe-2zxxr\" (UID: \"46944f21-f6bf-41fc-af89-392daeb9693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:47.709212 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.709179 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsldl\" (UniqueName: \"kubernetes.io/projected/46944f21-f6bf-41fc-af89-392daeb9693f-kube-api-access-nsldl\") pod \"maas-api-revoke-sub-e68ae2fe-2zxxr\" (UID: \"46944f21-f6bf-41fc-af89-392daeb9693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:47.709400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.709227 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e68ae2fe-2zxxr\" (UID: \"46944f21-f6bf-41fc-af89-392daeb9693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:47.709400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.709326 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e68ae2fe-2zxxr\" (UID: \"46944f21-f6bf-41fc-af89-392daeb9693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:47.709974 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.709943 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e68ae2fe-2zxxr\" (UID: \"46944f21-f6bf-41fc-af89-392daeb9693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:47.711766 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.711747 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e68ae2fe-2zxxr\" (UID: \"46944f21-f6bf-41fc-af89-392daeb9693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:47.733257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.733232 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsldl\" (UniqueName: \"kubernetes.io/projected/46944f21-f6bf-41fc-af89-392daeb9693f-kube-api-access-nsldl\") pod \"maas-api-revoke-sub-e68ae2fe-2zxxr\" (UID: \"46944f21-f6bf-41fc-af89-392daeb9693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:47.787846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:47.787819 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:48.095196 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:35:48.095155 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46944f21_f6bf_41fc_af89_392daeb9693f.slice/crio-a928a7db6c6dcc4fd564837e184ff6782facee9ddb1d2c818d025be59e471cef WatchSource:0}: Error finding container a928a7db6c6dcc4fd564837e184ff6782facee9ddb1d2c818d025be59e471cef: Status 404 returned error can't find the container with id a928a7db6c6dcc4fd564837e184ff6782facee9ddb1d2c818d025be59e471cef Oct 30 04:35:48.096931 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:48.096900 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr"] Oct 30 04:35:48.948725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:48.948692 4793 generic.go:356] "Generic (PLEG): container finished" podID="46944f21-f6bf-41fc-af89-392daeb9693f" containerID="47f7cc3e6bfa0f014370406cee1dbcc9ac15a4c30243bc1e8ac800f6216a2e48" exitCode=0 Oct 30 04:35:48.949052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:48.948761 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" event={"ID":"46944f21-f6bf-41fc-af89-392daeb9693f","Type":"ContainerDied","Data":"47f7cc3e6bfa0f014370406cee1dbcc9ac15a4c30243bc1e8ac800f6216a2e48"} Oct 30 04:35:48.949052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:48.948809 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" event={"ID":"46944f21-f6bf-41fc-af89-392daeb9693f","Type":"ContainerStarted","Data":"a928a7db6c6dcc4fd564837e184ff6782facee9ddb1d2c818d025be59e471cef"} Oct 30 04:35:49.114315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:49.114277 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:50.115857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.115828 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:50.237976 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.237904 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-xks-ca\") pod \"46944f21-f6bf-41fc-af89-392daeb9693f\" (UID: \"46944f21-f6bf-41fc-af89-392daeb9693f\") " Oct 30 04:35:50.238111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.238039 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-openshift-ca\") pod \"46944f21-f6bf-41fc-af89-392daeb9693f\" (UID: \"46944f21-f6bf-41fc-af89-392daeb9693f\") " Oct 30 04:35:50.238111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.238075 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46944f21-f6bf-41fc-af89-392daeb9693f-kube-api-access-nsldl\" (UniqueName: \"kubernetes.io/projected/46944f21-f6bf-41fc-af89-392daeb9693f-kube-api-access-nsldl\") pod \"46944f21-f6bf-41fc-af89-392daeb9693f\" (UID: \"46944f21-f6bf-41fc-af89-392daeb9693f\") " Oct 30 04:35:50.238465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.238440 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-openshift-ca" pod "46944f21-f6bf-41fc-af89-392daeb9693f" (UID: "46944f21-f6bf-41fc-af89-392daeb9693f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:35:50.238648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.238629 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:50.240017 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.239988 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-xks-ca" pod "46944f21-f6bf-41fc-af89-392daeb9693f" (UID: "46944f21-f6bf-41fc-af89-392daeb9693f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:35:50.240234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.240215 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46944f21-f6bf-41fc-af89-392daeb9693f-kube-api-access-nsldl" pod "46944f21-f6bf-41fc-af89-392daeb9693f" (UID: "46944f21-f6bf-41fc-af89-392daeb9693f"). InnerVolumeSpecName "kube-api-access-nsldl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:35:50.339894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.339850 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsldl\" (UniqueName: \"kubernetes.io/projected/46944f21-f6bf-41fc-af89-392daeb9693f-kube-api-access-nsldl\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:50.339894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.339890 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46944f21-f6bf-41fc-af89-392daeb9693f-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:35:50.788712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.788675 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:50.966436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.966388 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" Oct 30 04:35:50.966605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.966412 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr" event={"ID":"46944f21-f6bf-41fc-af89-392daeb9693f","Type":"ContainerDied","Data":"a928a7db6c6dcc4fd564837e184ff6782facee9ddb1d2c818d025be59e471cef"} Oct 30 04:35:50.966605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:50.966490 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a928a7db6c6dcc4fd564837e184ff6782facee9ddb1d2c818d025be59e471cef" Oct 30 04:35:52.477345 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:52.477309 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:53.319011 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.318975 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl"] Oct 30 04:35:53.319723 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.319706 4793 cpu_manager.go:414] "containerMap: removing container" podUID="46944f21-f6bf-41fc-af89-392daeb9693f" containerName="revoke-keys" Oct 30 04:35:53.319779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.319726 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46944f21-f6bf-41fc-af89-392daeb9693f" containerName="revoke-keys" Oct 30 04:35:53.319903 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.319891 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="46944f21-f6bf-41fc-af89-392daeb9693f" containerName="revoke-keys" Oct 30 04:35:53.323661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.323641 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.346717 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.346689 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl"] Oct 30 04:35:53.477578 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.477548 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.477578 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.477583 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.478029 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.477660 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f5d05a-24a1-4476-894e-ba216f49f782-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.478029 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.477771 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.478029 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.477818 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.478029 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.477896 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrj2l\" (UniqueName: \"kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-kube-api-access-nrj2l\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.478029 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.477939 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.478029 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.477978 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f5d05a-24a1-4476-894e-ba216f49f782-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.478254 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.478075 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.579492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.579381 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.580285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.580261 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.580846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.580826 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrj2l\" (UniqueName: \"kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-kube-api-access-nrj2l\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.581013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.580996 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.581144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.581131 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f5d05a-24a1-4476-894e-ba216f49f782-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.581528 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.581507 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.581683 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.581669 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.581786 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.581774 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.581942 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.581925 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f5d05a-24a1-4476-894e-ba216f49f782-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.582079 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.582051 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-data\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.582147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.580156 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-certs\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.585073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.582922 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.585073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.583155 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-credential-socket\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.585875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.585850 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f5d05a-24a1-4476-894e-ba216f49f782-istiod-ca-cert\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.591186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.591159 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f5d05a-24a1-4476-894e-ba216f49f782-istio-podinfo\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.592586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.592559 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-envoy\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.604347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.604311 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-istio-token\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.605819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.605797 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrj2l\" (UniqueName: \"kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-kube-api-access-nrj2l\") pod \"e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.641695 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.641673 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:53.965293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.965053 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl"] Oct 30 04:35:53.968532 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:35:53.968504 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f5d05a_24a1_4476_894e_ba216f49f782.slice/crio-ddf73d57d4e348c6658786520566b0c5c54efe3661a3008cfcee503448f3919c WatchSource:0}: Error finding container ddf73d57d4e348c6658786520566b0c5c54efe3661a3008cfcee503448f3919c: Status 404 returned error can't find the container with id ddf73d57d4e348c6658786520566b0c5c54efe3661a3008cfcee503448f3919c Oct 30 04:35:53.971045 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.970999 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:35:53.971121 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.971104 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:35:53.971173 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.971156 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:35:53.995445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:53.995401 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" event={"ID":"32f5d05a-24a1-4476-894e-ba216f49f782","Type":"ContainerStarted","Data":"ddf73d57d4e348c6658786520566b0c5c54efe3661a3008cfcee503448f3919c"} Oct 30 04:35:54.217199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:54.217096 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:54.478705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:54.478609 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:55.004741 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:55.004700 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" event={"ID":"32f5d05a-24a1-4476-894e-ba216f49f782","Type":"ContainerStarted","Data":"805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c"} Oct 30 04:35:55.082642 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:55.082573 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" podStartSLOduration=2.08255196 podStartE2EDuration="2.08255196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:35:55.067892795 +0000 UTC m=+2278.353767571" watchObservedRunningTime="2026-10-30 04:35:55.08255196 +0000 UTC m=+2278.368426736" Oct 30 04:35:55.641970 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:55.641932 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:35:55.643597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:55.643574 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 04:35:55.643671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:55.643622 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" podUID="32f5d05a-24a1-4476-894e-ba216f49f782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 04:35:55.926092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:55.926052 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:56.642512 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:56.642477 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 04:35:56.642919 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:56.642534 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" podUID="32f5d05a-24a1-4476-894e-ba216f49f782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 04:35:57.642696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:57.642652 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 04:35:57.643165 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:57.642704 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" podUID="32f5d05a-24a1-4476-894e-ba216f49f782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 04:35:57.718843 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:57.718802 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:58.642524 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:58.642440 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 04:35:58.642524 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:58.642505 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" podUID="32f5d05a-24a1-4476-894e-ba216f49f782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 04:35:59.414834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:59.412961 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:35:59.642597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:59.642564 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 04:35:59.642773 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:35:59.642624 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" podUID="32f5d05a-24a1-4476-894e-ba216f49f782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 04:36:00.642276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:00.642243 4793 patch_prober.go:29] interesting pod/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 04:36:00.642656 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:00.642303 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" podUID="32f5d05a-24a1-4476-894e-ba216f49f782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 04:36:01.105223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:01.105125 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:01.646934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:01.646907 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:36:01.647374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:01.647196 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:36:01.647890 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:01.647874 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:36:02.837774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:02.837737 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:04.486839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:04.486799 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:06.196235 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:06.196201 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:07.809818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:07.809716 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:09.639003 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:09.638895 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:09.955101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:09.955066 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:11.218132 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:36:11.218094 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:36:11.296409 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:11.296377 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:12.970791 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:12.970754 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:14.621699 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:14.621663 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:16.613455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:16.613397 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:18.426227 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:18.426188 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:20.276559 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:20.276522 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:21.914142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:21.914109 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:23.846307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:23.846273 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:25.615906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:25.615870 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:27.399294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:27.399261 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:28.043691 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:28.043656 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:28.323763 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:28.323680 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:29.178157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:29.178125 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:30.730739 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:30.730709 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:32.427407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:32.427370 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:34.125976 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:34.125941 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:35.903619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:35.903572 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:37.650262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:37.650224 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:39.322654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:39.322599 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:41.075997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:41.075958 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:41.218955 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:36:41.218922 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:36:42.749360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:42.749326 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:44.433454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:44.433397 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:46.104827 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:46.104795 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:47.782357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:47.782321 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:49.432436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:49.432389 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:51.046712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:51.046682 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:52.793189 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:52.793155 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:54.529569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:54.529536 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:54.750063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:54.750028 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:56.226212 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:56.226176 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:57.917848 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:57.917809 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:36:59.696187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:36:59.696146 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:01.310849 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:01.310809 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:03.010331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:03.010292 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:04.714562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:04.714526 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:06.516202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:06.516160 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:08.017668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:08.017631 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:09.788806 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:09.788776 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:11.218716 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:37:11.218683 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:37:11.398554 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:11.398514 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:13.125968 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:13.125930 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:14.611797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:14.611761 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:16.287168 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:16.287132 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:17.968957 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:17.968919 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:19.709339 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:19.709305 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:21.319735 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:21.319699 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:23.032095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:23.032056 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:24.611400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:24.611364 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:26.315472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:26.315411 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:27.881511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:27.881473 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:29.502358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:29.502321 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:31.114888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:31.114854 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:32.793673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:32.793587 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:34.423004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:34.422968 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:36.033092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:36.033058 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5"] Oct 30 04:37:36.045377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:36.045349 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef8543e5-jvtb5"] Oct 30 04:37:36.115165 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:36.115136 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:36.846049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:36.846019 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d338b3ca-77cc-4e7d-a1f4-202833a479df" path="/var/lib/kubelet/pods/d338b3ca-77cc-4e7d-a1f4-202833a479df/volumes" Oct 30 04:37:37.797591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:37.797557 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:39.421647 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:39.421612 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:41.118300 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:41.118268 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:41.218968 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:37:41.218931 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:37:42.811379 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:42.811342 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:44.414388 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:44.414353 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:46.134089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:46.134052 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:47.730356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:47.730316 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:47.944881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:47.944846 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:49.036586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:49.036554 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl"] Oct 30 04:37:49.049081 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:49.049054 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e535d631-gz9gl"] Oct 30 04:37:49.522869 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:49.522832 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:50.846242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:50.846208 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4057ead-26ed-4fd9-a18d-01cb76c495e4" path="/var/lib/kubelet/pods/b4057ead-26ed-4fd9-a18d-01cb76c495e4/volumes" Oct 30 04:37:51.222494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:51.222460 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:52.543414 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:52.543376 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4"] Oct 30 04:37:52.543927 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:52.543624 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerName="istio-proxy" containerID="cri-o://e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df" gracePeriod=30 Oct 30 04:37:52.927258 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:52.927230 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:54.684551 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:54.684512 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:54.916943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:54.916910 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:56.314665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:56.314629 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:57.812665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.812604 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:37:57.872565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.872537 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-kube-api-access-wwhtk\" (UniqueName: \"kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-kube-api-access-wwhtk\") pod \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " Oct 30 04:37:57.872767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.872612 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-credential-socket\") pod \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " Oct 30 04:37:57.872767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.872660 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-token\" (UniqueName: \"kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-token\") pod \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " Oct 30 04:37:57.872767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.872712 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-envoy\") pod \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " Oct 30 04:37:57.872962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.872798 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-data\") pod \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " Oct 30 04:37:57.872962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.872844 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-socket\") pod \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " Oct 30 04:37:57.872962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.872879 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-certs\") pod \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " Oct 30 04:37:57.873108 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.872982 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istiod-ca-cert\") pod \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " Oct 30 04:37:57.873714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.873039 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-credential-socket" pod "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" (UID: "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:37:57.873856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.873255 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-socket" pod "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" (UID: "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:37:57.873856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.873283 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-certs" pod "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" (UID: "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:37:57.873856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.873449 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-data" pod "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" (UID: "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:37:57.873856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.873610 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istiod-ca-cert" pod "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" (UID: "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:37:57.873856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.873115 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-podinfo\") pod \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\" (UID: \"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda\") " Oct 30 04:37:57.873856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.873598 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" event={"ID":"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda","Type":"ContainerDied","Data":"e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df"} Oct 30 04:37:57.873856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.873821 4793 scope.go:122] "RemoveContainer" containerID="e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df" Oct 30 04:37:57.874245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.873570 4793 generic.go:356] "Generic (PLEG): container finished" podID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerID="e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df" exitCode=0 Oct 30 04:37:57.874316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.873661 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" Oct 30 04:37:57.875850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.874693 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4" event={"ID":"932a7bf8-fd43-4de4-aa5c-1ba2fc545eda","Type":"ContainerDied","Data":"bc480f961b55af67dd8f986305b293de61b6f8e02c9ec724c9b6b49cdd6416cf"} Oct 30 04:37:57.875850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.875509 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:37:57.875850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.875528 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:37:57.875850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.875541 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:37:57.875850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.875554 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:37:57.875850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.875566 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:37:57.876570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.876544 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-kube-api-access-wwhtk" pod "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" (UID: "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda"). InnerVolumeSpecName "kube-api-access-wwhtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:37:57.879472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.879319 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-podinfo" pod "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" (UID: "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:37:57.879472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.879401 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-envoy" pod "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" (UID: "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:37:57.879637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.879511 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-token" pod "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" (UID: "932a7bf8-fd43-4de4-aa5c-1ba2fc545eda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:37:57.899898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.899826 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:37:57.902518 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.902501 4793 scope.go:122] "RemoveContainer" containerID="e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df" Oct 30 04:37:57.902803 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:37:57.902781 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df\": container with ID starting with e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df not found: ID does not exist" containerID="e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df" Oct 30 04:37:57.902899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.902806 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df"} err="failed to get container status \"e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df\": rpc error: code = NotFound desc = could not find container \"e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df\": container with ID starting with e95da8d872ec24efff89ea51c6a49d7422c3e89eddd60cb086cfaa4466eff1df not found: ID does not exist" Oct 30 04:37:57.976898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.976871 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:37:57.976898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.976892 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhtk\" (UniqueName: \"kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-kube-api-access-wwhtk\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:37:57.977098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.976905 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:37:57.977098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:57.976924 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:37:58.223747 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:58.223715 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4"] Oct 30 04:37:58.227833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:58.227810 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-5jxq4"] Oct 30 04:37:58.845805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:58.845766 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" path="/var/lib/kubelet/pods/932a7bf8-fd43-4de4-aa5c-1ba2fc545eda/volumes" Oct 30 04:37:59.519379 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:37:59.519343 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:00.031933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:00.031900 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz"] Oct 30 04:38:00.035147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:00.035119 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:00.044273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:00.044202 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ceb8735-mb2mz"] Oct 30 04:38:00.252310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:00.252274 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:00.849808 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:00.849774 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab1fd5b-73b3-4a6e-8d93-40333314269c" path="/var/lib/kubelet/pods/bab1fd5b-73b3-4a6e-8d93-40333314269c/volumes" Oct 30 04:38:01.161409 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:01.161321 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:01.946122 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:01.946092 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:02.160026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:02.159983 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:02.881065 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:02.881035 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:04.605361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:04.605318 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:05.240237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.240205 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw"] Oct 30 04:38:05.240757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.240742 4793 cpu_manager.go:414] "containerMap: removing container" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerName="istio-proxy" Oct 30 04:38:05.240816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.240758 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerName="istio-proxy" Oct 30 04:38:05.240853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.240831 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="932a7bf8-fd43-4de4-aa5c-1ba2fc545eda" containerName="istio-proxy" Oct 30 04:38:05.246106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.246087 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:05.262714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.262681 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw"] Oct 30 04:38:05.272442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.271091 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:38:05.272442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.271363 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 04:38:05.275305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.275263 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:38:05.275486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.275407 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:38:05.358161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.358130 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xr5d\" (UniqueName: \"kubernetes.io/projected/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-kube-api-access-2xr5d\") pod \"maas-api-revoke-sub-5d8bdbb3-4vbcw\" (UID: \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:05.358344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.358321 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d8bdbb3-4vbcw\" (UID: \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:05.358399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.358379 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d8bdbb3-4vbcw\" (UID: \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:05.459708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.459676 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d8bdbb3-4vbcw\" (UID: \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:05.459886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.459738 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d8bdbb3-4vbcw\" (UID: \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:05.459886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.459841 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xr5d\" (UniqueName: \"kubernetes.io/projected/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-kube-api-access-2xr5d\") pod \"maas-api-revoke-sub-5d8bdbb3-4vbcw\" (UID: \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:05.460538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.460513 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d8bdbb3-4vbcw\" (UID: \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:05.462271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.462247 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d8bdbb3-4vbcw\" (UID: \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:05.483891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.483870 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xr5d\" (UniqueName: \"kubernetes.io/projected/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-kube-api-access-2xr5d\") pod \"maas-api-revoke-sub-5d8bdbb3-4vbcw\" (UID: \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:05.560054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.560001 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:05.862296 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:38:05.862265 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef92d91_6d4f_42a6_ad9a_2d2bc6c481f0.slice/crio-994bac4656353519db4b992f8588b55e0d02b111028a96a1820a8b026a9ae3d7 WatchSource:0}: Error finding container 994bac4656353519db4b992f8588b55e0d02b111028a96a1820a8b026a9ae3d7: Status 404 returned error can't find the container with id 994bac4656353519db4b992f8588b55e0d02b111028a96a1820a8b026a9ae3d7 Oct 30 04:38:05.864076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.864044 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw"] Oct 30 04:38:05.927477 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:05.927452 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" event={"ID":"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0","Type":"ContainerStarted","Data":"994bac4656353519db4b992f8588b55e0d02b111028a96a1820a8b026a9ae3d7"} Oct 30 04:38:06.218125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:06.218089 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:06.935230 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:06.935158 4793 generic.go:356] "Generic (PLEG): container finished" podID="8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0" containerID="460618eab1b0b517128d5a6377e439d16147d7dea3c2f9c20a26dd6bda61da5c" exitCode=0 Oct 30 04:38:06.935600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:06.935240 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" event={"ID":"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0","Type":"ContainerDied","Data":"460618eab1b0b517128d5a6377e439d16147d7dea3c2f9c20a26dd6bda61da5c"} Oct 30 04:38:07.810487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:07.810446 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:08.104822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.104793 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:08.201932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.201904 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-xks-ca\") pod \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\" (UID: \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\") " Oct 30 04:38:08.202115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.201972 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-openshift-ca\") pod \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\" (UID: \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\") " Oct 30 04:38:08.202115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.202039 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-kube-api-access-2xr5d\" (UniqueName: \"kubernetes.io/projected/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-kube-api-access-2xr5d\") pod \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\" (UID: \"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0\") " Oct 30 04:38:08.202372 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.202342 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-openshift-ca" pod "8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0" (UID: "8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:38:08.202541 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.202521 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:38:08.204198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.204169 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-xks-ca" pod "8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0" (UID: "8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:38:08.204301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.204281 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-kube-api-access-2xr5d" pod "8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0" (UID: "8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0"). InnerVolumeSpecName "kube-api-access-2xr5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:38:08.303701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.303667 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:38:08.303701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.303696 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xr5d\" (UniqueName: \"kubernetes.io/projected/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0-kube-api-access-2xr5d\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:38:08.949616 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.949580 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" event={"ID":"8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0","Type":"ContainerDied","Data":"994bac4656353519db4b992f8588b55e0d02b111028a96a1820a8b026a9ae3d7"} Oct 30 04:38:08.949804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.949627 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="994bac4656353519db4b992f8588b55e0d02b111028a96a1820a8b026a9ae3d7" Oct 30 04:38:08.949804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:08.949590 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw" Oct 30 04:38:09.609893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:09.609856 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:11.219129 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:38:11.219091 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:38:11.299849 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:11.299810 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:13.008593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:13.008556 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:14.035056 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:14.035027 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88"] Oct 30 04:38:14.046602 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:14.046569 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e39b40-h7q88"] Oct 30 04:38:14.673526 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:14.673485 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:14.845348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:14.845313 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fe975f-76ed-4576-882d-9f5bdcd32b9a" path="/var/lib/kubelet/pods/04fe975f-76ed-4576-882d-9f5bdcd32b9a/volumes" Oct 30 04:38:16.307857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:16.307820 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:18.095522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:18.095485 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:19.732714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:19.732680 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:21.412313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:21.412277 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:23.040793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:23.040759 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:24.031395 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:24.031366 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g"] Oct 30 04:38:24.043787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:24.043761 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31797ec2-dvs2g"] Oct 30 04:38:24.706778 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:24.706745 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:24.846321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:24.846289 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1bfac90-d100-42fa-bfa7-60f4ece89fea" path="/var/lib/kubelet/pods/f1bfac90-d100-42fa-bfa7-60f4ece89fea/volumes" Oct 30 04:38:26.636377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:26.636336 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:27.900213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:27.900173 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:29.591655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:29.591614 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:31.301983 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:31.301948 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:32.993021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:32.992987 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:34.597933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:34.597894 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:36.294841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:36.294801 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:38.002832 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:38.002795 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:39.624225 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:39.624181 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:41.218400 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:38:41.218367 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:38:41.342817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:41.342784 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:43.117531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:43.117492 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:44.812924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:44.812890 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:46.528674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:46.528638 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:48.242005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:48.241968 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:49.824879 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:49.824843 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:51.622978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:51.622937 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:53.201553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:53.201518 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:54.343367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:54.343337 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w"] Oct 30 04:38:54.343821 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:54.343582 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" podUID="2e8b1b7d-8c0b-41ef-964f-42c406f53364" containerName="istio-proxy" containerID="cri-o://19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1" gracePeriod=30 Oct 30 04:38:54.860908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:54.860870 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:55.092917 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:55.092884 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:56.608493 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:56.608455 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:58.304357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:58.304319 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:38:58.423206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:58.423181 4793 scope.go:122] "RemoveContainer" containerID="d11b4337b3b1cefafb6d134bedde5653666abcff26e2cbd015cea17428c9ef3c" Oct 30 04:38:58.432786 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:58.432763 4793 scope.go:122] "RemoveContainer" containerID="b7ef2d148bbf2c263257d5d0bdf317a82614b25bc7bbd8ae77375c1fdf5b4d91" Oct 30 04:38:58.441498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:58.441481 4793 scope.go:122] "RemoveContainer" containerID="da6a5b2b67d04caf33cb1f893d1eea96f406b30c51a3bf064634c6227491d061" Oct 30 04:38:59.614992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.614930 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:38:59.666082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666045 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-credential-socket\") pod \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " Oct 30 04:38:59.666231 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666094 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-certs\") pod \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " Oct 30 04:38:59.666231 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666127 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istiod-ca-cert\") pod \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " Oct 30 04:38:59.666231 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666176 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-envoy\") pod \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " Oct 30 04:38:59.666409 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666241 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-kube-api-access-2lph4\" (UniqueName: \"kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-kube-api-access-2lph4\") pod \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " Oct 30 04:38:59.666409 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666276 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-credential-socket" pod "2e8b1b7d-8c0b-41ef-964f-42c406f53364" (UID: "2e8b1b7d-8c0b-41ef-964f-42c406f53364"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:38:59.666409 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666332 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-token\" (UniqueName: \"kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-token\") pod \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " Oct 30 04:38:59.666409 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666380 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-data\") pod \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " Oct 30 04:38:59.666661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666461 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-certs" pod "2e8b1b7d-8c0b-41ef-964f-42c406f53364" (UID: "2e8b1b7d-8c0b-41ef-964f-42c406f53364"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:38:59.666661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666559 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-podinfo\") pod \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " Oct 30 04:38:59.666661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666600 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-socket\") pod \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\" (UID: \"2e8b1b7d-8c0b-41ef-964f-42c406f53364\") " Oct 30 04:38:59.666818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666673 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istiod-ca-cert" pod "2e8b1b7d-8c0b-41ef-964f-42c406f53364" (UID: "2e8b1b7d-8c0b-41ef-964f-42c406f53364"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:38:59.667043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.666752 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-data" pod "2e8b1b7d-8c0b-41ef-964f-42c406f53364" (UID: "2e8b1b7d-8c0b-41ef-964f-42c406f53364"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:38:59.667043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.667014 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-socket" pod "2e8b1b7d-8c0b-41ef-964f-42c406f53364" (UID: "2e8b1b7d-8c0b-41ef-964f-42c406f53364"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:38:59.667198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.667066 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:38:59.667198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.667087 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:38:59.669040 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.669012 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-kube-api-access-2lph4" pod "2e8b1b7d-8c0b-41ef-964f-42c406f53364" (UID: "2e8b1b7d-8c0b-41ef-964f-42c406f53364"). InnerVolumeSpecName "kube-api-access-2lph4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:38:59.669152 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.669105 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-podinfo" pod "2e8b1b7d-8c0b-41ef-964f-42c406f53364" (UID: "2e8b1b7d-8c0b-41ef-964f-42c406f53364"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:38:59.671821 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.671798 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-envoy" pod "2e8b1b7d-8c0b-41ef-964f-42c406f53364" (UID: "2e8b1b7d-8c0b-41ef-964f-42c406f53364"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:38:59.671915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.671887 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-token" pod "2e8b1b7d-8c0b-41ef-964f-42c406f53364" (UID: "2e8b1b7d-8c0b-41ef-964f-42c406f53364"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:38:59.768756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.768728 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:38:59.768756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.768755 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:38:59.768939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.768764 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:38:59.768939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.768773 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:38:59.768939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.768783 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:38:59.768939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.768791 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e8b1b7d-8c0b-41ef-964f-42c406f53364-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:38:59.768939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.768799 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lph4\" (UniqueName: \"kubernetes.io/projected/2e8b1b7d-8c0b-41ef-964f-42c406f53364-kube-api-access-2lph4\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:38:59.926075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:38:59.926028 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:00.315352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:00.315274 4793 generic.go:356] "Generic (PLEG): container finished" podID="2e8b1b7d-8c0b-41ef-964f-42c406f53364" containerID="19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1" exitCode=0 Oct 30 04:39:00.315352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:00.315322 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" Oct 30 04:39:00.315583 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:00.315359 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" event={"ID":"2e8b1b7d-8c0b-41ef-964f-42c406f53364","Type":"ContainerDied","Data":"19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1"} Oct 30 04:39:00.315583 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:00.315427 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w" event={"ID":"2e8b1b7d-8c0b-41ef-964f-42c406f53364","Type":"ContainerDied","Data":"3ec1de42481d3b9058e6abae6c46606c736d92cb560ad3aa40058f1ccf08953e"} Oct 30 04:39:00.315583 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:00.315457 4793 scope.go:122] "RemoveContainer" containerID="19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1" Oct 30 04:39:00.328157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:00.328136 4793 scope.go:122] "RemoveContainer" containerID="19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1" Oct 30 04:39:00.328472 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:39:00.328444 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1\": container with ID starting with 19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1 not found: ID does not exist" containerID="19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1" Oct 30 04:39:00.328548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:00.328484 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1"} err="failed to get container status \"19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1\": rpc error: code = NotFound desc = could not find container \"19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1\": container with ID starting with 19860ad337bf2f7e8c8c549928c85e20e8f4564e1596efe805918e45b0d995d1 not found: ID does not exist" Oct 30 04:39:00.381941 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:00.381919 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w"] Oct 30 04:39:00.394809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:00.394784 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-swk8w"] Oct 30 04:39:00.847373 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:00.847340 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e8b1b7d-8c0b-41ef-964f-42c406f53364" path="/var/lib/kubelet/pods/2e8b1b7d-8c0b-41ef-964f-42c406f53364/volumes" Oct 30 04:39:01.654142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:01.654099 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:03.309169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.309139 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:03.540704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.540672 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:03.631530 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.631444 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729"] Oct 30 04:39:03.632155 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.632133 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2e8b1b7d-8c0b-41ef-964f-42c406f53364" containerName="istio-proxy" Oct 30 04:39:03.632155 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.632155 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e8b1b7d-8c0b-41ef-964f-42c406f53364" containerName="istio-proxy" Oct 30 04:39:03.632368 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.632188 4793 cpu_manager.go:414] "containerMap: removing container" podUID="8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0" containerName="revoke-keys" Oct 30 04:39:03.632368 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.632195 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0" containerName="revoke-keys" Oct 30 04:39:03.632368 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.632352 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0" containerName="revoke-keys" Oct 30 04:39:03.632568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.632376 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e8b1b7d-8c0b-41ef-964f-42c406f53364" containerName="istio-proxy" Oct 30 04:39:03.636979 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.636959 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.662767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.662740 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729"] Oct 30 04:39:03.670127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.670104 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-fcd6a6-openshift-default-dockercfg-n97h9\"" Oct 30 04:39:03.711148 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.711120 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds6cc\" (UniqueName: \"kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-kube-api-access-ds6cc\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.711274 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.711170 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-socket\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.711274 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.711190 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-podinfo\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.711398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.711358 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-credential-socket\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.711501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.711405 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-data\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.711575 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.711495 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-token\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.711644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.711625 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istiod-ca-cert\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.711737 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.711721 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-certs\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.711794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.711780 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-envoy\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.810439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.810385 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.812888 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istiod-ca-cert\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.812994 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-certs\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.813043 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-envoy\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.813080 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds6cc\" (UniqueName: \"kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-kube-api-access-ds6cc\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.813131 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-socket\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.813150 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-podinfo\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.813200 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-credential-socket\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.813230 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-data\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.813264 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-token\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.813405 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-certs\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.813684 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-socket\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.814944 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-data\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.814943 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-credential-socket\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.817101 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istiod-ca-cert\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.817646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.817477 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-envoy\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.818695 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.817808 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-podinfo\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.836883 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.836856 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-token\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.837821 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.837789 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds6cc\" (UniqueName: \"kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-kube-api-access-ds6cc\") pod \"e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:03.957918 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:03.957888 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:04.044486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:04.044445 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:04.276511 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:39:04.276477 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51f0c70_e6a7_4bf3_a367_511a9e2b9ccc.slice/crio-e3f2b648608f896f9bf4e2812084b8992153d12f509411ee8d19e25f6ef4af8c WatchSource:0}: Error finding container e3f2b648608f896f9bf4e2812084b8992153d12f509411ee8d19e25f6ef4af8c: Status 404 returned error can't find the container with id e3f2b648608f896f9bf4e2812084b8992153d12f509411ee8d19e25f6ef4af8c Oct 30 04:39:04.276923 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:04.276887 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729"] Oct 30 04:39:04.278977 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:04.278939 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:39:04.279066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:04.279010 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:39:04.279066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:04.279043 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:39:04.345004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:04.344971 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" event={"ID":"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc","Type":"ContainerStarted","Data":"e3f2b648608f896f9bf4e2812084b8992153d12f509411ee8d19e25f6ef4af8c"} Oct 30 04:39:05.003129 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:05.003068 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:05.353551 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:05.353460 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" event={"ID":"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc","Type":"ContainerStarted","Data":"88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639"} Oct 30 04:39:05.413047 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:05.412997 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" podStartSLOduration=2.41298221 podStartE2EDuration="2.41298221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:05.410878352 +0000 UTC m=+2468.696753152" watchObservedRunningTime="2026-10-30 04:39:05.41298221 +0000 UTC m=+2468.698856985" Oct 30 04:39:05.958058 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:05.958026 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:05.963546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:05.963524 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:06.361161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:06.361079 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:06.361986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:06.361963 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:06.624453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:06.624347 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:07.965759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:07.965725 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:08.186753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:08.186715 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:08.400334 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:08.400235 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:09.770500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.770385 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:09.824373 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.824336 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2"] Oct 30 04:39:09.828391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.828373 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" Oct 30 04:39:09.847406 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.847375 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2"] Oct 30 04:39:09.854279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.854250 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-fcd6a6\"" Oct 30 04:39:09.854279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.854264 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-fcd6a6-dockercfg-ldhkv\"" Oct 30 04:39:09.887835 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.886964 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpdqz\" (UniqueName: \"kubernetes.io/projected/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-kube-api-access-rpdqz\") pod \"payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2\" (UID: \"3a1419a8-5951-4bd3-945f-7f68f18cc7e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" Oct 30 04:39:09.887835 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.887117 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2\" (UID: \"3a1419a8-5951-4bd3-945f-7f68f18cc7e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" Oct 30 04:39:09.949512 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.949476 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz"] Oct 30 04:39:09.972548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.972518 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" Oct 30 04:39:09.976811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.976704 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz"] Oct 30 04:39:09.990190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.990158 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdqz\" (UniqueName: \"kubernetes.io/projected/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-kube-api-access-rpdqz\") pod \"payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2\" (UID: \"3a1419a8-5951-4bd3-945f-7f68f18cc7e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" Oct 30 04:39:09.990504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.990479 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2\" (UID: \"3a1419a8-5951-4bd3-945f-7f68f18cc7e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" Oct 30 04:39:09.992565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.992541 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2\" (UID: \"3a1419a8-5951-4bd3-945f-7f68f18cc7e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" Oct 30 04:39:09.996265 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:09.996225 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:10.014902 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:10.014877 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdqz\" (UniqueName: \"kubernetes.io/projected/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-kube-api-access-rpdqz\") pod \"payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2\" (UID: \"3a1419a8-5951-4bd3-945f-7f68f18cc7e6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" Oct 30 04:39:10.092248 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:10.092175 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtt8h\" (UniqueName: \"kubernetes.io/projected/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-kube-api-access-wtt8h\") pod \"payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz\" (UID: \"8ffbf8ef-592b-4543-9e5f-fe5b526e0281\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" Oct 30 04:39:10.092248 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:10.092212 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz\" (UID: \"8ffbf8ef-592b-4543-9e5f-fe5b526e0281\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" Oct 30 04:39:10.144528 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:10.144501 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" Oct 30 04:39:10.194883 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:10.193395 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz\" (UID: \"8ffbf8ef-592b-4543-9e5f-fe5b526e0281\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" Oct 30 04:39:10.194883 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:10.193691 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtt8h\" (UniqueName: \"kubernetes.io/projected/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-kube-api-access-wtt8h\") pod \"payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz\" (UID: \"8ffbf8ef-592b-4543-9e5f-fe5b526e0281\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" Oct 30 04:39:10.194883 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:10.194788 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz\" (UID: \"8ffbf8ef-592b-4543-9e5f-fe5b526e0281\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" Oct 30 04:39:10.218738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:10.218039 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtt8h\" (UniqueName: \"kubernetes.io/projected/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-kube-api-access-wtt8h\") pod \"payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz\" (UID: \"8ffbf8ef-592b-4543-9e5f-fe5b526e0281\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" Oct 30 04:39:10.218738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:10.218065 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:10.297934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:10.297889 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" Oct 30 04:39:10.487153 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:39:10.487121 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1419a8_5951_4bd3_945f_7f68f18cc7e6.slice/crio-acad8e947c16bfa3288b558e04be47ce798d37faef19e56ce93398cfc6e4a2e2 WatchSource:0}: Error finding container acad8e947c16bfa3288b558e04be47ce798d37faef19e56ce93398cfc6e4a2e2: Status 404 returned error can't find the container with id acad8e947c16bfa3288b558e04be47ce798d37faef19e56ce93398cfc6e4a2e2 Oct 30 04:39:10.487547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:10.487525 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2"] Oct 30 04:39:10.611562 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:39:10.611525 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffbf8ef_592b_4543_9e5f_fe5b526e0281.slice/crio-90e5fe1dbdb51e4b5bdc27ea71edc283ff8581f47edb0a3c077d47a08d698669 WatchSource:0}: Error finding container 90e5fe1dbdb51e4b5bdc27ea71edc283ff8581f47edb0a3c077d47a08d698669: Status 404 returned error can't find the container with id 90e5fe1dbdb51e4b5bdc27ea71edc283ff8581f47edb0a3c077d47a08d698669 Oct 30 04:39:10.611785 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:10.611763 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz"] Oct 30 04:39:11.218938 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:39:11.218900 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:39:11.400412 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:11.400380 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" event={"ID":"8ffbf8ef-592b-4543-9e5f-fe5b526e0281","Type":"ContainerStarted","Data":"7babdc1e8e780bc3c6ae97bb8cc0f244a1dec793dd799d4e6aef57ebf69d3c81"} Oct 30 04:39:11.400646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:11.400627 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" Oct 30 04:39:11.400707 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:11.400670 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" event={"ID":"8ffbf8ef-592b-4543-9e5f-fe5b526e0281","Type":"ContainerStarted","Data":"90e5fe1dbdb51e4b5bdc27ea71edc283ff8581f47edb0a3c077d47a08d698669"} Oct 30 04:39:11.402363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:11.402325 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" event={"ID":"3a1419a8-5951-4bd3-945f-7f68f18cc7e6","Type":"ContainerStarted","Data":"1d1ca75585a0bc534ab8c3c33730bb4a9f4e4d6420decab7019d2352eacad6e6"} Oct 30 04:39:11.402514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:11.402380 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" event={"ID":"3a1419a8-5951-4bd3-945f-7f68f18cc7e6","Type":"ContainerStarted","Data":"acad8e947c16bfa3288b558e04be47ce798d37faef19e56ce93398cfc6e4a2e2"} Oct 30 04:39:11.402514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:11.402446 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" Oct 30 04:39:11.442170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:11.442127 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" podStartSLOduration=2.442114272 podStartE2EDuration="2.442114272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:11.441676444 +0000 UTC m=+2474.727551220" watchObservedRunningTime="2026-10-30 04:39:11.442114272 +0000 UTC m=+2474.727989047" Oct 30 04:39:11.485851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:11.485756 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" podStartSLOduration=2.485738456 podStartE2EDuration="2.485738456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:11.48219437 +0000 UTC m=+2474.768069144" watchObservedRunningTime="2026-10-30 04:39:11.485738456 +0000 UTC m=+2474.771613233" Oct 30 04:39:11.913384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:11.913290 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:12.035519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:12.035487 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh"] Oct 30 04:39:12.047841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:12.047813 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afcc50b1-5hkrh"] Oct 30 04:39:12.846662 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:12.846623 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13d97113-2871-4d8b-a2ba-8b6cb0272160" path="/var/lib/kubelet/pods/13d97113-2871-4d8b-a2ba-8b6cb0272160/volumes" Oct 30 04:39:13.565661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:13.565626 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:15.125389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:15.125349 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:16.741063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:16.741025 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:18.179934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.179904 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2"] Oct 30 04:39:18.184007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.183989 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.212806 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.212774 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 04:39:18.212963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.212847 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-fcd6a6-serving-cert\"" Oct 30 04:39:18.212963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.212887 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 04:39:18.212963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.212931 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gw5bk\"" Oct 30 04:39:18.213132 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.213034 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 04:39:18.216296 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.216277 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2"] Oct 30 04:39:18.288210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.288180 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.288357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.288224 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a585f87f-a29f-4928-a8b2-ca69834ac97d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.288357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.288300 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.288357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.288318 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb77h\" (UniqueName: \"kubernetes.io/projected/a585f87f-a29f-4928-a8b2-ca69834ac97d-kube-api-access-jb77h\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.384705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.384677 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:18.390040 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.390009 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.390194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.390063 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a585f87f-a29f-4928-a8b2-ca69834ac97d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.390194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.390108 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.390194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.390162 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb77h\" (UniqueName: \"kubernetes.io/projected/a585f87f-a29f-4928-a8b2-ca69834ac97d-kube-api-access-jb77h\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.390607 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:39:18.390583 4793 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-serving-cert: secret "maas-api-e2e-shared-a-w2-fcd6a6-serving-cert" not found Oct 30 04:39:18.390747 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:39:18.390660 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-maas-api-tls podName:a585f87f-a29f-4928-a8b2-ca69834ac97d nodeName:}" failed. No retries permitted until 2026-10-30 04:39:18.890641705 +0000 UTC m=+2482.176516458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-maas-api-tls") pod "maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" (UID: "a585f87f-a29f-4928-a8b2-ca69834ac97d") : secret "maas-api-e2e-shared-a-w2-fcd6a6-serving-cert" not found Oct 30 04:39:18.391067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.391043 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a585f87f-a29f-4928-a8b2-ca69834ac97d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.392886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.392865 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.414393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.414368 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb77h\" (UniqueName: \"kubernetes.io/projected/a585f87f-a29f-4928-a8b2-ca69834ac97d-kube-api-access-jb77h\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.595743 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.595662 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:18.818757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.818721 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:18.896744 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.896677 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:18.899160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:18.899135 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:19.097682 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:19.097647 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:19.413202 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:39:19.413171 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda585f87f_a29f_4928_a8b2_ca69834ac97d.slice/crio-6c48fd2e4eed15ae8d9a8af731c74eb017e9cf8119702a33eb6d1a1a3c75de07 WatchSource:0}: Error finding container 6c48fd2e4eed15ae8d9a8af731c74eb017e9cf8119702a33eb6d1a1a3c75de07: Status 404 returned error can't find the container with id 6c48fd2e4eed15ae8d9a8af731c74eb017e9cf8119702a33eb6d1a1a3c75de07 Oct 30 04:39:19.414796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:19.414777 4793 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:39:19.415197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:19.415173 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2"] Oct 30 04:39:19.460881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:19.460838 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" event={"ID":"a585f87f-a29f-4928-a8b2-ca69834ac97d","Type":"ContainerStarted","Data":"6c48fd2e4eed15ae8d9a8af731c74eb017e9cf8119702a33eb6d1a1a3c75de07"} Oct 30 04:39:20.124184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:20.124151 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:20.471362 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:20.471329 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" event={"ID":"a585f87f-a29f-4928-a8b2-ca69834ac97d","Type":"ContainerStarted","Data":"87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf"} Oct 30 04:39:20.471780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:20.471403 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:20.520149 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:20.520090 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" podStartSLOduration=2.035598621 podStartE2EDuration="2.520073402s" totalImagesPullingTime="484.474781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:20.515720878 +0000 UTC m=+2483.801595678" watchObservedRunningTime="2026-10-30 04:39:20.520073402 +0000 UTC m=+2483.805948178" Oct 30 04:39:21.822560 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:21.822527 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:22.412805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:22.412773 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" Oct 30 04:39:22.413023 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:22.412904 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" Oct 30 04:39:23.528895 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:23.528861 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:25.218956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:25.218921 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:26.487506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:26.487473 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 04:39:27.128835 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:27.128794 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:28.817777 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:28.817736 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:29.885979 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:29.885944 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:30.370464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:30.370409 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:30.598483 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:30.593203 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:30.804013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:30.803980 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk"] Oct 30 04:39:30.804305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:30.804277 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" podUID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" containerName="istio-proxy" containerID="cri-o://e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70" gracePeriod=30 Oct 30 04:39:30.816847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:30.816814 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd"] Oct 30 04:39:30.817043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:30.817021 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" podUID="f2c0eabb-3612-403a-a5ae-24fbae4c3930" containerName="istio-proxy" containerID="cri-o://d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1" gracePeriod=30 Oct 30 04:39:30.841089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:30.840953 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:32.300286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:32.300248 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:33.179359 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:33.179324 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:33.678242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:33.678201 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:33.952330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:33.952245 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:34.290959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:34.290783 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:34.513542 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:34.513503 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:34.721338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:34.721301 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:35.583549 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.583467 4793 generic.go:356] "Generic (PLEG): container finished" podID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerID="88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639" exitCode=0 Oct 30 04:39:35.583549 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.583536 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" event={"ID":"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc","Type":"ContainerDied","Data":"88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639"} Oct 30 04:39:35.584016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.583992 4793 scope.go:122] "RemoveContainer" containerID="88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639" Oct 30 04:39:35.585648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.585613 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:39:35.585732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.585705 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:39:35.585772 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.585749 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:39:35.632997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.632965 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:35.877441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.875753 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:35.888469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.887184 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9"] Oct 30 04:39:35.892181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.892155 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" Oct 30 04:39:35.898150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.898124 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9"] Oct 30 04:39:35.914821 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.914793 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-e070ea-dockercfg-xzbsv\"" Oct 30 04:39:35.914963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:35.914791 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e070ea\"" Oct 30 04:39:36.004958 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.004657 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgb8n\" (UniqueName: \"kubernetes.io/projected/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-kube-api-access-cgb8n\") pod \"payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9\" (UID: \"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" Oct 30 04:39:36.004958 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.004813 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9\" (UID: \"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" Oct 30 04:39:36.106811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.106719 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9\" (UID: \"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" Oct 30 04:39:36.106920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.106884 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgb8n\" (UniqueName: \"kubernetes.io/projected/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-kube-api-access-cgb8n\") pod \"payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9\" (UID: \"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" Oct 30 04:39:36.109637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.108074 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9\" (UID: \"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" Oct 30 04:39:36.121469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.118953 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:39:36.121469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.120298 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:36.134735 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.132277 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgb8n\" (UniqueName: \"kubernetes.io/projected/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-kube-api-access-cgb8n\") pod \"payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9\" (UID: \"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" Oct 30 04:39:36.208389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.207310 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-certs\") pod \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " Oct 30 04:39:36.208389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.207636 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istiod-ca-cert\") pod \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " Oct 30 04:39:36.208389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.207676 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-envoy\") pod \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " Oct 30 04:39:36.208389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.207708 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-certs" pod "dcd66ef7-63fb-42ee-8302-5f04a5c90274" (UID: "dcd66ef7-63fb-42ee-8302-5f04a5c90274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:39:36.208389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.207734 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-data\") pod \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " Oct 30 04:39:36.208389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.207826 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-credential-socket\") pod \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " Oct 30 04:39:36.208389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.207856 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-socket\") pod \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " Oct 30 04:39:36.208389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.207934 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-token\" (UniqueName: \"kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-token\") pod \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " Oct 30 04:39:36.208389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.207983 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-podinfo\") pod \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " Oct 30 04:39:36.208389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.208025 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istiod-ca-cert" pod "dcd66ef7-63fb-42ee-8302-5f04a5c90274" (UID: "dcd66ef7-63fb-42ee-8302-5f04a5c90274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:39:36.208389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.208035 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-kube-api-access-tl55d\" (UniqueName: \"kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-kube-api-access-tl55d\") pod \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\" (UID: \"dcd66ef7-63fb-42ee-8302-5f04a5c90274\") " Oct 30 04:39:36.208389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.208244 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-credential-socket" pod "dcd66ef7-63fb-42ee-8302-5f04a5c90274" (UID: "dcd66ef7-63fb-42ee-8302-5f04a5c90274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:39:36.209272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.208546 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.209272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.208568 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.209272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.208582 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.210107 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.210025 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-data" pod "dcd66ef7-63fb-42ee-8302-5f04a5c90274" (UID: "dcd66ef7-63fb-42ee-8302-5f04a5c90274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:39:36.210322 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.210302 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-socket" pod "dcd66ef7-63fb-42ee-8302-5f04a5c90274" (UID: "dcd66ef7-63fb-42ee-8302-5f04a5c90274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:39:36.210948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.210924 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-envoy" pod "dcd66ef7-63fb-42ee-8302-5f04a5c90274" (UID: "dcd66ef7-63fb-42ee-8302-5f04a5c90274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:39:36.211731 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.211713 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:39:36.212174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.211938 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-podinfo" pod "dcd66ef7-63fb-42ee-8302-5f04a5c90274" (UID: "dcd66ef7-63fb-42ee-8302-5f04a5c90274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:39:36.212370 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.212351 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-kube-api-access-tl55d" pod "dcd66ef7-63fb-42ee-8302-5f04a5c90274" (UID: "dcd66ef7-63fb-42ee-8302-5f04a5c90274"). InnerVolumeSpecName "kube-api-access-tl55d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:39:36.213920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.213896 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-token" pod "dcd66ef7-63fb-42ee-8302-5f04a5c90274" (UID: "dcd66ef7-63fb-42ee-8302-5f04a5c90274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:39:36.290409 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.290369 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" Oct 30 04:39:36.309798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.309766 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-token\" (UniqueName: \"kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-token\") pod \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " Oct 30 04:39:36.309942 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.309814 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-data\") pod \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " Oct 30 04:39:36.309942 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.309921 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-kube-api-access-fbjjx\" (UniqueName: \"kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-kube-api-access-fbjjx\") pod \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " Oct 30 04:39:36.310070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.309956 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-certs\") pod \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " Oct 30 04:39:36.310070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.310002 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istiod-ca-cert\") pod \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " Oct 30 04:39:36.310070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.310047 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-podinfo\") pod \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " Oct 30 04:39:36.310239 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.310081 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-socket\") pod \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " Oct 30 04:39:36.310239 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.310112 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-envoy\") pod \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " Oct 30 04:39:36.310239 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.310218 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-credential-socket\") pod \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\" (UID: \"f2c0eabb-3612-403a-a5ae-24fbae4c3930\") " Oct 30 04:39:36.310953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.310684 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.310953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.310689 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-certs" pod "f2c0eabb-3612-403a-a5ae-24fbae4c3930" (UID: "f2c0eabb-3612-403a-a5ae-24fbae4c3930"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:39:36.310953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.310710 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.310953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.310729 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.310953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.310743 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl55d\" (UniqueName: \"kubernetes.io/projected/dcd66ef7-63fb-42ee-8302-5f04a5c90274-kube-api-access-tl55d\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.310953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.310759 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.310953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.310772 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd66ef7-63fb-42ee-8302-5f04a5c90274-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.311383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.311080 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istiod-ca-cert" pod "f2c0eabb-3612-403a-a5ae-24fbae4c3930" (UID: "f2c0eabb-3612-403a-a5ae-24fbae4c3930"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:39:36.311509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.311468 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-socket" pod "f2c0eabb-3612-403a-a5ae-24fbae4c3930" (UID: "f2c0eabb-3612-403a-a5ae-24fbae4c3930"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:39:36.311644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.311505 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-credential-socket" pod "f2c0eabb-3612-403a-a5ae-24fbae4c3930" (UID: "f2c0eabb-3612-403a-a5ae-24fbae4c3930"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:39:36.312120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.312095 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-data" pod "f2c0eabb-3612-403a-a5ae-24fbae4c3930" (UID: "f2c0eabb-3612-403a-a5ae-24fbae4c3930"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:39:36.314328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.314297 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-podinfo" pod "f2c0eabb-3612-403a-a5ae-24fbae4c3930" (UID: "f2c0eabb-3612-403a-a5ae-24fbae4c3930"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:39:36.314474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.314382 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-kube-api-access-fbjjx" pod "f2c0eabb-3612-403a-a5ae-24fbae4c3930" (UID: "f2c0eabb-3612-403a-a5ae-24fbae4c3930"). InnerVolumeSpecName "kube-api-access-fbjjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:39:36.315004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.314979 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-token" pod "f2c0eabb-3612-403a-a5ae-24fbae4c3930" (UID: "f2c0eabb-3612-403a-a5ae-24fbae4c3930"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:39:36.315242 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.315218 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-envoy" pod "f2c0eabb-3612-403a-a5ae-24fbae4c3930" (UID: "f2c0eabb-3612-403a-a5ae-24fbae4c3930"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:39:36.375256 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.372527 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:36.412514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.411893 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.412514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.411928 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.412514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.411945 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbjjx\" (UniqueName: \"kubernetes.io/projected/f2c0eabb-3612-403a-a5ae-24fbae4c3930-kube-api-access-fbjjx\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.412514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.411963 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.412514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.411979 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.412514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.411993 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.412514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.412008 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.412514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.412023 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.412514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.412038 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2c0eabb-3612-403a-a5ae-24fbae4c3930-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:39:36.595594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.595559 4793 generic.go:356] "Generic (PLEG): container finished" podID="f2c0eabb-3612-403a-a5ae-24fbae4c3930" containerID="d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1" exitCode=0 Oct 30 04:39:36.595752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.595611 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" Oct 30 04:39:36.595752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.595661 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" event={"ID":"f2c0eabb-3612-403a-a5ae-24fbae4c3930","Type":"ContainerDied","Data":"d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1"} Oct 30 04:39:36.595752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.595733 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd" event={"ID":"f2c0eabb-3612-403a-a5ae-24fbae4c3930","Type":"ContainerDied","Data":"10aa0afd6d832956bc281418fabac483fc78d721ab59252d6c82d9d60e2e59f2"} Oct 30 04:39:36.595893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.595763 4793 scope.go:122] "RemoveContainer" containerID="d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1" Oct 30 04:39:36.598042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.598018 4793 generic.go:356] "Generic (PLEG): container finished" podID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" containerID="e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70" exitCode=0 Oct 30 04:39:36.598184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.598114 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" event={"ID":"dcd66ef7-63fb-42ee-8302-5f04a5c90274","Type":"ContainerDied","Data":"e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70"} Oct 30 04:39:36.598184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.598158 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" event={"ID":"dcd66ef7-63fb-42ee-8302-5f04a5c90274","Type":"ContainerDied","Data":"6d2048f6ff27443666ccc1f3a917e1ec0fff813237c404944df67894535859ae"} Oct 30 04:39:36.598314 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.598236 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk" Oct 30 04:39:36.601588 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.601562 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" event={"ID":"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc","Type":"ContainerStarted","Data":"f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad"} Oct 30 04:39:36.607221 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.607204 4793 scope.go:122] "RemoveContainer" containerID="d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1" Oct 30 04:39:36.607517 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:39:36.607493 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1\": container with ID starting with d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1 not found: ID does not exist" containerID="d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1" Oct 30 04:39:36.607571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.607530 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1"} err="failed to get container status \"d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1\": rpc error: code = NotFound desc = could not find container \"d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1\": container with ID starting with d5cd77c44933a191b736d29bd8b8a6692b9e005f8adc312e57a8bcd6575e28a1 not found: ID does not exist" Oct 30 04:39:36.607571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.607552 4793 scope.go:122] "RemoveContainer" containerID="e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70" Oct 30 04:39:36.615775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.615757 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9"] Oct 30 04:39:36.619861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.619838 4793 scope.go:122] "RemoveContainer" containerID="e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70" Oct 30 04:39:36.620110 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:39:36.620092 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70\": container with ID starting with e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70 not found: ID does not exist" containerID="e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70" Oct 30 04:39:36.620193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.620117 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70"} err="failed to get container status \"e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70\": rpc error: code = NotFound desc = could not find container \"e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70\": container with ID starting with e25dcedcf48b72fab81a0d749c568e6aecfd349469938ba636d9744517265c70 not found: ID does not exist" Oct 30 04:39:36.621337 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:39:36.621315 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc4dabd_cfc2_4a79_aadc_28833d9f99ef.slice/crio-37712b97f2e2240dcf89534d46ede8e6f1c8d6a157cc2b2e23f6fb8122300d0b WatchSource:0}: Error finding container 37712b97f2e2240dcf89534d46ede8e6f1c8d6a157cc2b2e23f6fb8122300d0b: Status 404 returned error can't find the container with id 37712b97f2e2240dcf89534d46ede8e6f1c8d6a157cc2b2e23f6fb8122300d0b Oct 30 04:39:36.717516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.717484 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk"] Oct 30 04:39:36.721400 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.721377 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-9sbtk"] Oct 30 04:39:36.795332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.795296 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd"] Oct 30 04:39:36.798268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.798239 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-36bbac-openshift-default-6f9cf6c5cf-fjhhd"] Oct 30 04:39:36.845768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.845735 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" path="/var/lib/kubelet/pods/dcd66ef7-63fb-42ee-8302-5f04a5c90274/volumes" Oct 30 04:39:36.846215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.846200 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c0eabb-3612-403a-a5ae-24fbae4c3930" path="/var/lib/kubelet/pods/f2c0eabb-3612-403a-a5ae-24fbae4c3930/volumes" Oct 30 04:39:36.963451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.960654 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:36.963451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.960798 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 04:39:36.963451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:36.960840 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 04:39:37.610594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:37.610541 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" event={"ID":"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef","Type":"ContainerStarted","Data":"649a207cb58c46f5079f762937aca98d8736d13a16893424b0d8fd19de5897b1"} Oct 30 04:39:37.610805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:37.610723 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" Oct 30 04:39:37.610805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:37.610760 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" event={"ID":"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef","Type":"ContainerStarted","Data":"37712b97f2e2240dcf89534d46ede8e6f1c8d6a157cc2b2e23f6fb8122300d0b"} Oct 30 04:39:37.651600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:37.651553 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" podStartSLOduration=2.651540582 podStartE2EDuration="2.651540582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:37.650960044 +0000 UTC m=+2500.936834819" watchObservedRunningTime="2026-10-30 04:39:37.651540582 +0000 UTC m=+2500.937415357" Oct 30 04:39:37.900557 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:37.900464 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:37.982442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:37.982177 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 04:39:37.982442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:37.982235 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 04:39:38.056441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.048541 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn"] Oct 30 04:39:38.078442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.069072 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ed8a9f-njvnn"] Oct 30 04:39:38.116044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.116006 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:38.327608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.327571 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:38.593652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.593558 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:38.846475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.846396 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83183572-0667-4816-88b4-4f85b8aa041a" path="/var/lib/kubelet/pods/83183572-0667-4816-88b4-4f85b8aa041a/volumes" Oct 30 04:39:38.894366 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.894328 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2"] Oct 30 04:39:38.895275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.895245 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f2c0eabb-3612-403a-a5ae-24fbae4c3930" containerName="istio-proxy" Oct 30 04:39:38.895275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.895274 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c0eabb-3612-403a-a5ae-24fbae4c3930" containerName="istio-proxy" Oct 30 04:39:38.895491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.895295 4793 cpu_manager.go:414] "containerMap: removing container" podUID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" containerName="istio-proxy" Oct 30 04:39:38.895491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.895303 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" containerName="istio-proxy" Oct 30 04:39:38.895867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.895842 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c0eabb-3612-403a-a5ae-24fbae4c3930" containerName="istio-proxy" Oct 30 04:39:38.895962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.895878 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd66ef7-63fb-42ee-8302-5f04a5c90274" containerName="istio-proxy" Oct 30 04:39:38.899619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.899594 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:38.924442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.919753 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-e6b837-openshift-default-dockercfg-9lwnd\"" Oct 30 04:39:38.925267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.925242 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2"] Oct 30 04:39:38.945927 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.945900 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-certs\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:38.946049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.945937 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-socket\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:38.946049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.945974 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-credential-socket\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:38.946213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.946196 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-envoy\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:38.946321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.946245 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-podinfo\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:38.946484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.946317 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istiod-ca-cert\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:38.946484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.946384 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-token\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:38.946484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.946470 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-data\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:38.946643 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.946504 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq48f\" (UniqueName: \"kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-kube-api-access-jq48f\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:38.984031 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.983880 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 04:39:38.984031 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:38.983938 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 04:39:39.051956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.051919 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-envoy\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.088441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.084814 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-podinfo\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.088441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.084868 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istiod-ca-cert\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.088441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.084902 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-token\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.088441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.084954 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-data\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.088441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.084992 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq48f\" (UniqueName: \"kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-kube-api-access-jq48f\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.088441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.085101 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-certs\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.088441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.085129 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-socket\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.088441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.085167 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-credential-socket\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.088441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.085633 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-credential-socket\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.088441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.052704 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:39.089493 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.089465 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-data\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.089661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.083879 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-envoy\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.090456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.090434 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istiod-ca-cert\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.091157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.091133 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-certs\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.091834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.091811 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-socket\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.117534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.103053 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-podinfo\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.120266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.120242 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-token\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.120592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.120563 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq48f\" (UniqueName: \"kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-kube-api-access-jq48f\") pod \"e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.219676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.219645 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:39.306786 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.306705 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:39.589144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.589111 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2"] Oct 30 04:39:39.596709 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:39:39.596679 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abcfdfb_40e7_441d_bff1_c74858fe5aaf.slice/crio-0729b28c3d29589ef526628a404a4dbccdb2835bba78a3757a5ed58aff60a60d WatchSource:0}: Error finding container 0729b28c3d29589ef526628a404a4dbccdb2835bba78a3757a5ed58aff60a60d: Status 404 returned error can't find the container with id 0729b28c3d29589ef526628a404a4dbccdb2835bba78a3757a5ed58aff60a60d Oct 30 04:39:39.599321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.599282 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:39:39.599391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.599370 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:39:39.599464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.599428 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:39:39.631351 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.631317 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" event={"ID":"3abcfdfb-40e7-441d-bff1-c74858fe5aaf","Type":"ContainerStarted","Data":"0729b28c3d29589ef526628a404a4dbccdb2835bba78a3757a5ed58aff60a60d"} Oct 30 04:39:39.773510 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.767947 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:39.787999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.787970 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t"] Oct 30 04:39:39.793446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.791409 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" Oct 30 04:39:39.817389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.815814 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-66217b-dockercfg-4h5l5\"" Oct 30 04:39:39.817389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.816665 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-66217b\"" Oct 30 04:39:39.822330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.821817 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t"] Oct 30 04:39:39.907507 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.905673 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402fa222-73dc-4da9-960e-56661d43ddbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t\" (UID: \"402fa222-73dc-4da9-960e-56661d43ddbb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" Oct 30 04:39:39.907507 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.905799 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm5pb\" (UniqueName: \"kubernetes.io/projected/402fa222-73dc-4da9-960e-56661d43ddbb-kube-api-access-jm5pb\") pod \"payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t\" (UID: \"402fa222-73dc-4da9-960e-56661d43ddbb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" Oct 30 04:39:39.953347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.953306 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f"] Oct 30 04:39:39.983224 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.983189 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 04:39:40.017516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.017481 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 04:39:40.018041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.016915 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:40.018214 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.017553 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm5pb\" (UniqueName: \"kubernetes.io/projected/402fa222-73dc-4da9-960e-56661d43ddbb-kube-api-access-jm5pb\") pod \"payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t\" (UID: \"402fa222-73dc-4da9-960e-56661d43ddbb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" Oct 30 04:39:40.019591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:39.984374 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" Oct 30 04:39:40.019591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.019067 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402fa222-73dc-4da9-960e-56661d43ddbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t\" (UID: \"402fa222-73dc-4da9-960e-56661d43ddbb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" Oct 30 04:39:40.020042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.019995 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402fa222-73dc-4da9-960e-56661d43ddbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t\" (UID: \"402fa222-73dc-4da9-960e-56661d43ddbb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" Oct 30 04:39:40.021710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.021689 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f"] Oct 30 04:39:40.040935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.040898 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm5pb\" (UniqueName: \"kubernetes.io/projected/402fa222-73dc-4da9-960e-56661d43ddbb-kube-api-access-jm5pb\") pod \"payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t\" (UID: \"402fa222-73dc-4da9-960e-56661d43ddbb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" Oct 30 04:39:40.120794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.120319 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e83024f-5869-4dcd-868a-8c2afe7ff9af-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f\" (UID: \"3e83024f-5869-4dcd-868a-8c2afe7ff9af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" Oct 30 04:39:40.120794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.120519 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6b4\" (UniqueName: \"kubernetes.io/projected/3e83024f-5869-4dcd-868a-8c2afe7ff9af-kube-api-access-fd6b4\") pod \"payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f\" (UID: \"3e83024f-5869-4dcd-868a-8c2afe7ff9af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" Oct 30 04:39:40.132294 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.132269 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" Oct 30 04:39:40.223527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.222471 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e83024f-5869-4dcd-868a-8c2afe7ff9af-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f\" (UID: \"3e83024f-5869-4dcd-868a-8c2afe7ff9af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" Oct 30 04:39:40.223527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.222554 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6b4\" (UniqueName: \"kubernetes.io/projected/3e83024f-5869-4dcd-868a-8c2afe7ff9af-kube-api-access-fd6b4\") pod \"payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f\" (UID: \"3e83024f-5869-4dcd-868a-8c2afe7ff9af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" Oct 30 04:39:40.223527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.223483 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e83024f-5869-4dcd-868a-8c2afe7ff9af-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f\" (UID: \"3e83024f-5869-4dcd-868a-8c2afe7ff9af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" Oct 30 04:39:40.249066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.249023 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6b4\" (UniqueName: \"kubernetes.io/projected/3e83024f-5869-4dcd-868a-8c2afe7ff9af-kube-api-access-fd6b4\") pod \"payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f\" (UID: \"3e83024f-5869-4dcd-868a-8c2afe7ff9af\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" Oct 30 04:39:40.331962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.331766 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" Oct 30 04:39:40.494579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.493871 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t"] Oct 30 04:39:40.641206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.641176 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" event={"ID":"3abcfdfb-40e7-441d-bff1-c74858fe5aaf","Type":"ContainerStarted","Data":"5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e"} Oct 30 04:39:40.642887 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.642862 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" event={"ID":"402fa222-73dc-4da9-960e-56661d43ddbb","Type":"ContainerStarted","Data":"3be3b0e05877f32a92f93a57e0ae710d92c2de1a6b7f3082b15c2e5b37bdfe7a"} Oct 30 04:39:40.642988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.642906 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" event={"ID":"402fa222-73dc-4da9-960e-56661d43ddbb","Type":"ContainerStarted","Data":"709ddece5744201693cd22ba543d2158c7e998f359ea09ed92ddcf47d6a0784e"} Oct 30 04:39:40.642988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.642926 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" Oct 30 04:39:40.690442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.690122 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f"] Oct 30 04:39:40.692746 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:39:40.692711 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e83024f_5869_4dcd_868a_8c2afe7ff9af.slice/crio-0133a24b163fc892af2ec07fc7c7c9de386e1a448bbcf82b922a1b213167381f WatchSource:0}: Error finding container 0133a24b163fc892af2ec07fc7c7c9de386e1a448bbcf82b922a1b213167381f: Status 404 returned error can't find the container with id 0133a24b163fc892af2ec07fc7c7c9de386e1a448bbcf82b922a1b213167381f Oct 30 04:39:40.704916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.704872 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" podStartSLOduration=2.704858263 podStartE2EDuration="2.704858263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:40.702001412 +0000 UTC m=+2503.987876188" watchObservedRunningTime="2026-10-30 04:39:40.704858263 +0000 UTC m=+2503.990733037" Oct 30 04:39:40.976443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.961715 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 04:39:40.976443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:40.961773 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 04:39:41.220952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.220919 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:41.221770 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:39:41.221744 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:39:41.226259 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.226238 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:41.280444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.275860 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" podStartSLOduration=2.275838907 podStartE2EDuration="2.275838907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:40.743190707 +0000 UTC m=+2504.029065487" watchObservedRunningTime="2026-10-30 04:39:41.275838907 +0000 UTC m=+2504.561713684" Oct 30 04:39:41.313818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.313785 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:41.651333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.651242 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" event={"ID":"3e83024f-5869-4dcd-868a-8c2afe7ff9af","Type":"ContainerStarted","Data":"a547086e3872fcb553f015b06936bf6912019308b9a9d95e89975b11bbaeb44f"} Oct 30 04:39:41.651333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.651298 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" event={"ID":"3e83024f-5869-4dcd-868a-8c2afe7ff9af","Type":"ContainerStarted","Data":"0133a24b163fc892af2ec07fc7c7c9de386e1a448bbcf82b922a1b213167381f"} Oct 30 04:39:41.651751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.651727 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:41.651898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.651775 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" Oct 30 04:39:41.652350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.652330 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:39:41.696715 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.696648 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" podStartSLOduration=2.696629433 podStartE2EDuration="2.696629433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:41.69313426 +0000 UTC m=+2504.979009061" watchObservedRunningTime="2026-10-30 04:39:41.696629433 +0000 UTC m=+2504.982504209" Oct 30 04:39:41.963790 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.963759 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:41.964167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.964148 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:41.964850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:41.964828 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 04:39:43.121612 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:43.121579 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:43.369253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:43.369216 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:44.958887 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:44.958849 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:45.192570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:45.192526 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:46.668724 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.668690 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn"] Oct 30 04:39:46.672072 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.672050 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:46.700219 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.700193 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-e070ea-serving-cert\"" Oct 30 04:39:46.706402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.706373 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn"] Oct 30 04:39:46.710243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.710204 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:46.710387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.710367 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:46.710521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.710491 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqtk6\" (UniqueName: \"kubernetes.io/projected/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-kube-api-access-jqtk6\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:46.710600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.710554 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:46.811300 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.811267 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:46.811484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.811391 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:46.811484 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:39:46.811407 4793 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-serving-cert: secret "maas-api-e2e-shared-a-w3-e070ea-serving-cert" not found Oct 30 04:39:46.811630 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:39:46.811503 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-maas-api-tls podName:b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b nodeName:}" failed. No retries permitted until 2026-10-30 04:39:47.311480498 +0000 UTC m=+2510.597355269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-maas-api-tls") pod "maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" (UID: "b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b") : secret "maas-api-e2e-shared-a-w3-e070ea-serving-cert" not found Oct 30 04:39:46.811630 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.811526 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqtk6\" (UniqueName: \"kubernetes.io/projected/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-kube-api-access-jqtk6\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:46.811630 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.811575 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:46.812223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.812202 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:46.813863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.813828 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:46.835191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.835165 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqtk6\" (UniqueName: \"kubernetes.io/projected/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-kube-api-access-jqtk6\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:46.885464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:46.884750 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:47.108797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:47.108763 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:47.317351 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:47.317314 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:47.319701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:47.319679 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:47.330012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:47.329983 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:47.586751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:47.586717 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:47.894760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:47.894731 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn"] Oct 30 04:39:47.895939 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:39:47.895909 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7a9e7c7_6603_4e72_9159_0a2cc7d5f04b.slice/crio-3eebb4a667442c49481b6672e5e3e6d3b9d78ac233c53dbcb037210712edb51a WatchSource:0}: Error finding container 3eebb4a667442c49481b6672e5e3e6d3b9d78ac233c53dbcb037210712edb51a: Status 404 returned error can't find the container with id 3eebb4a667442c49481b6672e5e3e6d3b9d78ac233c53dbcb037210712edb51a Oct 30 04:39:48.603071 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:48.602976 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:48.622380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:48.622356 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" Oct 30 04:39:48.711269 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:48.711235 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" event={"ID":"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b","Type":"ContainerStarted","Data":"096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81"} Oct 30 04:39:48.711443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:48.711289 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" event={"ID":"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b","Type":"ContainerStarted","Data":"3eebb4a667442c49481b6672e5e3e6d3b9d78ac233c53dbcb037210712edb51a"} Oct 30 04:39:48.711443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:48.711317 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:48.756941 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:48.756881 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" podStartSLOduration=2.450791448 podStartE2EDuration="2.756868147s" totalImagesPullingTime="306.076699ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:39:48.754816705 +0000 UTC m=+2512.040691479" watchObservedRunningTime="2026-10-30 04:39:48.756868147 +0000 UTC m=+2512.042742922" Oct 30 04:39:50.402260 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:50.402223 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:51.655204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:51.655165 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" Oct 30 04:39:52.044527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:52.044493 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:52.265055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:52.265016 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:52.663765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:52.663737 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" Oct 30 04:39:53.754950 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:53.754908 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:54.724414 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:54.724387 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 04:39:54.980034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:54.979943 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:55.482814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:55.482776 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:57.204241 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:57.204195 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:58.476500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.476470 4793 scope.go:122] "RemoveContainer" containerID="218c69f6b1f8b1f6128ee7f51eb592968a2f1505b43217d6a99af0d76d588e6c" Oct 30 04:39:58.486665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.486639 4793 scope.go:122] "RemoveContainer" containerID="89c4db37ad53f6890a1f2c45ffbc096b68345ea385be23f3340fe14ebb2e2c72" Oct 30 04:39:58.677441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.677345 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr"] Oct 30 04:39:58.687436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.685290 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:58.703066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.702671 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-e6b837-serving-cert\"" Oct 30 04:39:58.704767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.704743 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr"] Oct 30 04:39:58.756653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.756572 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:58.756806 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.756654 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:58.756806 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.756717 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjdfh\" (UniqueName: \"kubernetes.io/projected/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-kube-api-access-kjdfh\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:58.756806 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.756790 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:58.856892 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.856593 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:58.858190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.858157 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:58.858341 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.858229 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:58.858341 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.858296 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjdfh\" (UniqueName: \"kubernetes.io/projected/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-kube-api-access-kjdfh\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:58.858341 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:39:58.858314 4793 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-serving-cert: secret "maas-api-e2e-shared-a-w0-e6b837-serving-cert" not found Oct 30 04:39:58.858555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.858342 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:58.858555 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:39:58.858378 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-maas-api-tls podName:721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35 nodeName:}" failed. No retries permitted until 2026-10-30 04:39:59.358356776 +0000 UTC m=+2522.644231531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-maas-api-tls") pod "maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" (UID: "721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35") : secret "maas-api-e2e-shared-a-w0-e6b837-serving-cert" not found Oct 30 04:39:58.859330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.859299 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:58.861649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.861621 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:58.884243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:58.884215 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjdfh\" (UniqueName: \"kubernetes.io/projected/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-kube-api-access-kjdfh\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:59.074870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:59.074784 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:39:59.367168 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:59.367082 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:59.369606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:59.369578 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:59.606712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:59.606674 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:39:59.922576 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:39:59.922536 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721e8c36_82b0_4a8c_8f0e_5a4ad1b77d35.slice/crio-313188782be34809d4898c36619e83bd0f6946b0679c80b03513dd36aa53ddec WatchSource:0}: Error finding container 313188782be34809d4898c36619e83bd0f6946b0679c80b03513dd36aa53ddec: Status 404 returned error can't find the container with id 313188782be34809d4898c36619e83bd0f6946b0679c80b03513dd36aa53ddec Oct 30 04:39:59.922576 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:39:59.922548 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr"] Oct 30 04:40:00.703885 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:00.703849 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:00.802279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:00.802243 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" event={"ID":"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35","Type":"ContainerStarted","Data":"2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84"} Oct 30 04:40:00.802519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:00.802298 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" event={"ID":"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35","Type":"ContainerStarted","Data":"313188782be34809d4898c36619e83bd0f6946b0679c80b03513dd36aa53ddec"} Oct 30 04:40:00.802519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:00.802345 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:40:00.849048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:00.849004 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" podStartSLOduration=2.549646244 podStartE2EDuration="2.848990349s" totalImagesPullingTime="299.344105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:00.846789006 +0000 UTC m=+2524.132663794" watchObservedRunningTime="2026-10-30 04:40:00.848990349 +0000 UTC m=+2524.134865160" Oct 30 04:40:02.485944 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:02.485909 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:04.228940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:04.228888 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:05.861150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:05.861116 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:06.816125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:06.816093 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 04:40:07.590952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:07.590914 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:09.233880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:09.233837 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:09.448719 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:09.448680 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:09.788487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:09.788451 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:10.249241 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:10.249203 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:10.519758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:10.519675 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:11.023575 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:11.023538 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:11.220096 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:40:11.220039 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:40:12.825380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:12.825339 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:13.415621 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:13.415582 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:14.115479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:14.115405 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:14.352231 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:14.352190 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:14.819432 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:14.819394 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:15.840715 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:15.840679 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:16.133662 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:16.107782 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:16.788787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:16.788730 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:17.780387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:17.780351 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:18.414509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:18.414482 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:20.199187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:20.199143 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:21.786316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:21.786277 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:22.975584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:22.975543 4793 generic.go:356] "Generic (PLEG): container finished" podID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerID="5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e" exitCode=0 Oct 30 04:40:22.976363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:22.975711 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" event={"ID":"3abcfdfb-40e7-441d-bff1-c74858fe5aaf","Type":"ContainerDied","Data":"5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e"} Oct 30 04:40:22.976363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:22.976144 4793 scope.go:122] "RemoveContainer" containerID="5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e" Oct 30 04:40:22.977836 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:22.977804 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:40:22.977911 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:22.977875 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:40:22.977958 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:22.977910 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:40:23.032381 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:23.032355 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl"] Oct 30 04:40:23.035802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:23.035778 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ba18b8d-dgjsl"] Oct 30 04:40:23.503824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:23.503782 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:23.853279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:23.853202 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:23.993184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:23.993146 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" event={"ID":"3abcfdfb-40e7-441d-bff1-c74858fe5aaf","Type":"ContainerStarted","Data":"d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411"} Oct 30 04:40:24.220986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.220954 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:40:24.222052 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.222023 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 04:40:24.222187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.222075 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 04:40:24.282953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.282917 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:24.606334 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.606239 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh"] Oct 30 04:40:24.610702 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.610676 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.632479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.632457 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh"] Oct 30 04:40:24.639527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.639502 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-247158-openshift-default-dockercfg-rvw8n\"" Oct 30 04:40:24.690629 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.690591 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-credential-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.690844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.690655 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.690844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.690735 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-data\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.690844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.690767 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae86f996-7f99-491c-b774-8b4b97dea5ee-istiod-ca-cert\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.690844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.690837 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-podinfo\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.691080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.690949 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-token\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.691080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.691004 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-certs\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.691080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.691052 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-envoy\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.691247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.691096 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9hrf\" (UniqueName: \"kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-kube-api-access-h9hrf\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.782117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.782080 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:24.794442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.791992 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-token\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.794442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.792059 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-certs\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.794442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.792115 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-envoy\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.794442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.792161 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9hrf\" (UniqueName: \"kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-kube-api-access-h9hrf\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.794442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.792226 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-credential-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.794442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.792273 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.794442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.792302 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-data\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.794442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.792328 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae86f996-7f99-491c-b774-8b4b97dea5ee-istiod-ca-cert\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.794442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.792444 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-podinfo\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.794442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.794091 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-certs\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.810437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.810156 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.822462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.810809 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-credential-socket\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.822462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.811465 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae86f996-7f99-491c-b774-8b4b97dea5ee-istiod-ca-cert\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.822462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.811834 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-data\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.822462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.816694 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-envoy\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.831764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.827014 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-token\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.831764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.830444 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-podinfo\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.835555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.835527 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9hrf\" (UniqueName: \"kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-kube-api-access-h9hrf\") pod \"e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.857494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.852684 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a28ba79-5e26-4973-b99d-5b97dc185d61" path="/var/lib/kubelet/pods/0a28ba79-5e26-4973-b99d-5b97dc185d61/volumes" Oct 30 04:40:24.935285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.935208 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:24.999155 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:24.999088 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:25.220735 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:25.220703 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 04:40:25.220943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:25.220767 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 04:40:25.234272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:25.234235 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:25.302105 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:40:25.302071 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae86f996_7f99_491c_b774_8b4b97dea5ee.slice/crio-4faa656bb0a633abcb934dc91ff7b62be5ab28ac687e97cd243ad7d5a8a377c2 WatchSource:0}: Error finding container 4faa656bb0a633abcb934dc91ff7b62be5ab28ac687e97cd243ad7d5a8a377c2: Status 404 returned error can't find the container with id 4faa656bb0a633abcb934dc91ff7b62be5ab28ac687e97cd243ad7d5a8a377c2 Oct 30 04:40:25.302876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:25.302853 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh"] Oct 30 04:40:25.304596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:25.304568 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:40:25.304674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:25.304629 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:40:25.304674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:25.304661 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:40:26.010600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:26.010562 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" event={"ID":"ae86f996-7f99-491c-b774-8b4b97dea5ee","Type":"ContainerStarted","Data":"f3f8f92ed658fb381b227ac0304def009e7d48d25d5e711d6c8dff6e6ee7d40f"} Oct 30 04:40:26.011039 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:26.010614 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" event={"ID":"ae86f996-7f99-491c-b774-8b4b97dea5ee","Type":"ContainerStarted","Data":"4faa656bb0a633abcb934dc91ff7b62be5ab28ac687e97cd243ad7d5a8a377c2"} Oct 30 04:40:26.221884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:26.221848 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 04:40:26.222180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:26.222150 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 04:40:26.652211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:26.652181 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:40:26.935830 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:26.935788 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:26.942861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:26.942834 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:26.986439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:26.986152 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:27.018035 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:27.017976 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" podStartSLOduration=3.017959956 podStartE2EDuration="3.017959956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:40:26.074659789 +0000 UTC m=+2549.360534579" watchObservedRunningTime="2026-10-30 04:40:27.017959956 +0000 UTC m=+2550.303834734" Oct 30 04:40:27.019177 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:27.019150 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:27.020043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:27.020019 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:40:27.220553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:27.220469 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 04:40:27.220553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:27.220524 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 04:40:28.220237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:28.220196 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 04:40:28.220680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:28.220275 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 04:40:28.602346 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:28.602265 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:28.842131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:28.842096 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:29.221206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:29.221170 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Oct 30 04:40:29.221645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:29.221224 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Oct 30 04:40:30.226696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:30.226664 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:40:30.228442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:30.228392 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 04:40:30.433175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:30.433138 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:30.664855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:30.664764 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:30.876970 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:30.876935 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:31.103647 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:31.103607 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:32.604965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:32.604929 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:34.203470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:34.203398 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:35.889474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:35.889441 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:37.032312 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:37.032275 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9"] Oct 30 04:40:37.044641 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:37.044613 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e939b9a-4h8g9"] Oct 30 04:40:37.414654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:37.414575 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:37.630184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:37.630153 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:38.847675 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:38.847641 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09401b03-488c-4420-9f0a-d8e6c7ce6aef" path="/var/lib/kubelet/pods/09401b03-488c-4420-9f0a-d8e6c7ce6aef/volumes" Oct 30 04:40:39.116161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:39.116080 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:40.729376 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:40.729292 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:41.218799 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:40:41.218767 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:40:42.325203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:42.325170 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:44.031680 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:44.031623 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:45.711303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:45.711258 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:47.385964 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:47.385929 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:47.605315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:47.605276 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:49.104073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:49.104038 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:50.619763 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:50.619720 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:50.834529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:50.834492 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:51.032724 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:51.032688 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr"] Oct 30 04:40:51.080718 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:51.080678 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:51.081361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:51.081333 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e68ae2fe-2zxxr"] Oct 30 04:40:51.314319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:51.314229 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:52.519508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:52.519465 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:52.850166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:52.850091 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46944f21-f6bf-41fc-af89-392daeb9693f" path="/var/lib/kubelet/pods/46944f21-f6bf-41fc-af89-392daeb9693f/volumes" Oct 30 04:40:54.363832 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:54.363797 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:54.588522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:54.588470 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:54.815396 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:54.815357 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:55.102789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:55.102700 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:56.453348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:56.453311 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:56.683572 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:56.683535 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:56.920437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:56.920328 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:57.351558 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:57.351521 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:58.310775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:58.310738 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:40:58.534525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:58.534495 4793 scope.go:122] "RemoveContainer" containerID="db2edba6f3e429f1e125a93020cf826c1fcc3799584b39e54bc6f2000c610299" Oct 30 04:40:58.544430 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:40:58.544396 4793 scope.go:122] "RemoveContainer" containerID="4548450670fa751210dda46e0f962a945e74cd9cbd20dd8ea1e6b209858ccb1c" Oct 30 04:41:00.084021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:00.083985 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:01.715199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:01.715162 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:03.383476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:03.383429 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:05.075442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:05.073931 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:06.729403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:06.729364 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:08.391508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:08.391469 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:10.053640 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:10.053599 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:11.219757 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:41:11.219716 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:41:11.908264 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:11.908232 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:12.020399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:12.020350 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 30 04:41:12.020612 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:12.020401 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 30 04:41:12.383567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:12.383530 4793 generic.go:356] "Generic (PLEG): container finished" podID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerID="f3f8f92ed658fb381b227ac0304def009e7d48d25d5e711d6c8dff6e6ee7d40f" exitCode=0 Oct 30 04:41:12.383976 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:12.383607 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" event={"ID":"ae86f996-7f99-491c-b774-8b4b97dea5ee","Type":"ContainerDied","Data":"f3f8f92ed658fb381b227ac0304def009e7d48d25d5e711d6c8dff6e6ee7d40f"} Oct 30 04:41:12.384051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:12.384034 4793 scope.go:122] "RemoveContainer" containerID="f3f8f92ed658fb381b227ac0304def009e7d48d25d5e711d6c8dff6e6ee7d40f" Oct 30 04:41:12.385890 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:12.385856 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:41:12.385992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:12.385956 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:41:12.386049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:12.386007 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:41:13.392474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:13.392433 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" event={"ID":"ae86f996-7f99-491c-b774-8b4b97dea5ee","Type":"ContainerStarted","Data":"d8bf977300ac8be66c18c46874d345f263b403eaa09636344bf9f74809dbb559"} Oct 30 04:41:13.724888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:13.724851 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:13.935873 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:13.935837 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:41:13.936199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:13.936036 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 30 04:41:13.936199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:13.936085 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 30 04:41:14.269032 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:14.268941 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:14.935840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:14.935802 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 30 04:41:14.936247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:14.935865 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 30 04:41:15.274701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.274607 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9"] Oct 30 04:41:15.282808 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.282781 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.287850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.287452 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9"] Oct 30 04:41:15.299923 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.299903 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-4c2415-serving-cert\"" Oct 30 04:41:15.344917 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.344886 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.345101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.344951 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7s6\" (UniqueName: \"kubernetes.io/projected/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-kube-api-access-8c7s6\") pod \"maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.345101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.344996 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.345101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.345027 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.446469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.446408 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7s6\" (UniqueName: \"kubernetes.io/projected/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-kube-api-access-8c7s6\") pod \"maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.446646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.446509 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.446646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.446560 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.446803 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.446705 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.447166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.447133 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.449005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.448978 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.449172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.449150 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.482714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.479860 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:15.482873 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.482717 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7s6\" (UniqueName: \"kubernetes.io/projected/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-kube-api-access-8c7s6\") pod \"maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.600700 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.600607 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:15.706780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.706334 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:15.931323 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.931292 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9"] Oct 30 04:41:15.931685 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:41:15.931650 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eaaa85c_519d_4004_8e11_2d46b5dddcfb.slice/crio-702a9e1b925547a98c012c5d0af1e5760a253f81774f068c7fe319f26fdb8073 WatchSource:0}: Error finding container 702a9e1b925547a98c012c5d0af1e5760a253f81774f068c7fe319f26fdb8073: Status 404 returned error can't find the container with id 702a9e1b925547a98c012c5d0af1e5760a253f81774f068c7fe319f26fdb8073 Oct 30 04:41:15.936500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.936474 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 30 04:41:15.936831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:15.936535 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 30 04:41:16.423209 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:16.423169 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" event={"ID":"3eaaa85c-519d-4004-8e11-2d46b5dddcfb","Type":"ContainerStarted","Data":"a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db"} Oct 30 04:41:16.423484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:16.423469 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:16.423615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:16.423603 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" event={"ID":"3eaaa85c-519d-4004-8e11-2d46b5dddcfb","Type":"ContainerStarted","Data":"702a9e1b925547a98c012c5d0af1e5760a253f81774f068c7fe319f26fdb8073"} Oct 30 04:41:16.500057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:16.499966 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" podStartSLOduration=1.194125982 podStartE2EDuration="1.499952087s" totalImagesPullingTime="305.826105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:41:16.496571066 +0000 UTC m=+2599.782445843" watchObservedRunningTime="2026-10-30 04:41:16.499952087 +0000 UTC m=+2599.785826876" Oct 30 04:41:16.508393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:16.508317 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:16.935781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:16.935752 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 30 04:41:16.935948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:16.935806 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 30 04:41:17.285060 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:17.284981 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:17.935597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:17.935567 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body= Oct 30 04:41:17.935773 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:17.935626 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" Oct 30 04:41:18.940309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:18.940279 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:41:18.940720 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:18.940623 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:41:18.941518 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:18.941496 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 04:41:19.118552 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:19.118520 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:20.336286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.336250 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:20.626997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.626917 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6"] Oct 30 04:41:20.630240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.630215 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.655553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.655520 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6"] Oct 30 04:41:20.656198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.656173 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-certs\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.656333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.656239 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138e743d-b5cc-4304-a424-e52af2ffd818-istio-podinfo\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.656333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.656284 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-envoy\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.656475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.656446 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-credential-socket\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.656545 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.656530 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfb5\" (UniqueName: \"kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-kube-api-access-gnfb5\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.656604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.656571 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-istio-token\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.656661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.656619 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-socket\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.656661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.656650 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138e743d-b5cc-4304-a424-e52af2ffd818-istiod-ca-cert\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.656789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.656701 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-data\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.659950 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.659923 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-20a650-openshift-default-dockercfg-lvcp2\"" Oct 30 04:41:20.678909 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.678880 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.760196 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-certs\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.760290 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138e743d-b5cc-4304-a424-e52af2ffd818-istio-podinfo\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.760335 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-envoy\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.760406 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-credential-socket\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.760496 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfb5\" (UniqueName: \"kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-kube-api-access-gnfb5\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.760537 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-istio-token\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.760586 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-socket\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.760621 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138e743d-b5cc-4304-a424-e52af2ffd818-istiod-ca-cert\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.760704 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-data\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.760869 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-certs\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.761081 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-data\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.763791 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-socket\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.764752 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138e743d-b5cc-4304-a424-e52af2ffd818-istiod-ca-cert\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.779444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.764987 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-credential-socket\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.804596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.803956 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-istio-token\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.808967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.808170 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138e743d-b5cc-4304-a424-e52af2ffd818-istio-podinfo\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.842373 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.840966 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-envoy\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.850626 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.850590 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfb5\" (UniqueName: \"kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-kube-api-access-gnfb5\") pod \"e2e-explicit-20a650-openshift-default-54678c495c-j52w6\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:20.900209 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.900120 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:20.946606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:20.946574 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:21.132311 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:21.132272 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:21.265218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:21.265178 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6"] Oct 30 04:41:21.265544 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:41:21.265519 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138e743d_b5cc_4304_a424_e52af2ffd818.slice/crio-4205161c0429b43e5b2910d422547770f6cf1b673faeb31f5df86dab0891a34c WatchSource:0}: Error finding container 4205161c0429b43e5b2910d422547770f6cf1b673faeb31f5df86dab0891a34c: Status 404 returned error can't find the container with id 4205161c0429b43e5b2910d422547770f6cf1b673faeb31f5df86dab0891a34c Oct 30 04:41:21.268361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:21.268318 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:41:21.268506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:21.268406 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:41:21.268506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:21.268469 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:41:21.461516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:21.461468 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" event={"ID":"138e743d-b5cc-4304-a424-e52af2ffd818","Type":"ContainerStarted","Data":"3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e"} Oct 30 04:41:21.462009 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:21.461536 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" event={"ID":"138e743d-b5cc-4304-a424-e52af2ffd818","Type":"ContainerStarted","Data":"4205161c0429b43e5b2910d422547770f6cf1b673faeb31f5df86dab0891a34c"} Oct 30 04:41:21.509669 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:21.509619 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" podStartSLOduration=1.509605821 podStartE2EDuration="1.509605821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:41:21.506188266 +0000 UTC m=+2604.792063042" watchObservedRunningTime="2026-10-30 04:41:21.509605821 +0000 UTC m=+2604.795480596" Oct 30 04:41:21.946978 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:21.946945 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:22.435207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:22.435172 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 04:41:22.440981 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:22.440941 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:22.952524 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:22.952496 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:23.479797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:23.479770 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:23.480801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:23.480775 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:41:23.972314 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:23.972277 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:24.127368 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:24.127327 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl"] Oct 30 04:41:24.127617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:24.127590 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" podUID="32f5d05a-24a1-4476-894e-ba216f49f782" containerName="istio-proxy" containerID="cri-o://805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c" gracePeriod=30 Oct 30 04:41:24.179410 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:24.179375 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:24.400667 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:24.400575 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:25.663456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:25.660552 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:25.918153 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:25.918039 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:26.217225 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:26.217193 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:28.091331 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:28.091294 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:29.412178 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.412153 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:41:29.484609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.484565 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-certs\") pod \"32f5d05a-24a1-4476-894e-ba216f49f782\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " Oct 30 04:41:29.484793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.484623 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-envoy\") pod \"32f5d05a-24a1-4476-894e-ba216f49f782\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " Oct 30 04:41:29.484793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.484656 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-credential-socket\") pod \"32f5d05a-24a1-4476-894e-ba216f49f782\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " Oct 30 04:41:29.484793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.484692 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f5d05a-24a1-4476-894e-ba216f49f782-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f5d05a-24a1-4476-894e-ba216f49f782-istiod-ca-cert\") pod \"32f5d05a-24a1-4476-894e-ba216f49f782\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " Oct 30 04:41:29.484793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.484751 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-istio-token\" (UniqueName: \"kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-istio-token\") pod \"32f5d05a-24a1-4476-894e-ba216f49f782\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " Oct 30 04:41:29.485037 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.484795 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32f5d05a-24a1-4476-894e-ba216f49f782-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f5d05a-24a1-4476-894e-ba216f49f782-istio-podinfo\") pod \"32f5d05a-24a1-4476-894e-ba216f49f782\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " Oct 30 04:41:29.485037 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.484831 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-data\") pod \"32f5d05a-24a1-4476-894e-ba216f49f782\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " Oct 30 04:41:29.485037 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.484970 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-kube-api-access-nrj2l\" (UniqueName: \"kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-kube-api-access-nrj2l\") pod \"32f5d05a-24a1-4476-894e-ba216f49f782\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " Oct 30 04:41:29.485204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.485038 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-socket\") pod \"32f5d05a-24a1-4476-894e-ba216f49f782\" (UID: \"32f5d05a-24a1-4476-894e-ba216f49f782\") " Oct 30 04:41:29.486078 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.486040 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-socket" pod "32f5d05a-24a1-4476-894e-ba216f49f782" (UID: "32f5d05a-24a1-4476-894e-ba216f49f782"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:41:29.486685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.486297 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-certs" pod "32f5d05a-24a1-4476-894e-ba216f49f782" (UID: "32f5d05a-24a1-4476-894e-ba216f49f782"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:41:29.486685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.486647 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-data" pod "32f5d05a-24a1-4476-894e-ba216f49f782" (UID: "32f5d05a-24a1-4476-894e-ba216f49f782"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:41:29.487393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.487360 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f5d05a-24a1-4476-894e-ba216f49f782-istiod-ca-cert" pod "32f5d05a-24a1-4476-894e-ba216f49f782" (UID: "32f5d05a-24a1-4476-894e-ba216f49f782"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:41:29.488016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.487995 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-credential-socket" pod "32f5d05a-24a1-4476-894e-ba216f49f782" (UID: "32f5d05a-24a1-4476-894e-ba216f49f782"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:41:29.489250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.489223 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32f5d05a-24a1-4476-894e-ba216f49f782-istio-podinfo" pod "32f5d05a-24a1-4476-894e-ba216f49f782" (UID: "32f5d05a-24a1-4476-894e-ba216f49f782"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:41:29.489593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.489560 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-kube-api-access-nrj2l" pod "32f5d05a-24a1-4476-894e-ba216f49f782" (UID: "32f5d05a-24a1-4476-894e-ba216f49f782"). InnerVolumeSpecName "kube-api-access-nrj2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:41:29.490063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.490039 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-istio-token" pod "32f5d05a-24a1-4476-894e-ba216f49f782" (UID: "32f5d05a-24a1-4476-894e-ba216f49f782"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:41:29.490237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.490219 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-envoy" pod "32f5d05a-24a1-4476-894e-ba216f49f782" (UID: "32f5d05a-24a1-4476-894e-ba216f49f782"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:41:29.527439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.527399 4793 generic.go:356] "Generic (PLEG): container finished" podID="32f5d05a-24a1-4476-894e-ba216f49f782" containerID="805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c" exitCode=0 Oct 30 04:41:29.527574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.527483 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" Oct 30 04:41:29.527574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.527485 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" event={"ID":"32f5d05a-24a1-4476-894e-ba216f49f782","Type":"ContainerDied","Data":"805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c"} Oct 30 04:41:29.527574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.527561 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl" event={"ID":"32f5d05a-24a1-4476-894e-ba216f49f782","Type":"ContainerDied","Data":"ddf73d57d4e348c6658786520566b0c5c54efe3661a3008cfcee503448f3919c"} Oct 30 04:41:29.527718 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.527582 4793 scope.go:122] "RemoveContainer" containerID="805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c" Oct 30 04:41:29.537981 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.537963 4793 scope.go:122] "RemoveContainer" containerID="805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c" Oct 30 04:41:29.538211 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:41:29.538194 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c\": container with ID starting with 805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c not found: ID does not exist" containerID="805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c" Oct 30 04:41:29.538259 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.538220 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c"} err="failed to get container status \"805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c\": rpc error: code = NotFound desc = could not find container \"805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c\": container with ID starting with 805174dec27d20b3b3758e976790e32d655975f9fbe38079eb8a1761b5d13c2c not found: ID does not exist" Oct 30 04:41:29.584031 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.584008 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl"] Oct 30 04:41:29.587384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.587360 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:41:29.587384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.587386 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:41:29.587585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.587409 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:41:29.587585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.587439 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f5d05a-24a1-4476-894e-ba216f49f782-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:41:29.587585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.587460 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:41:29.587585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.587471 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f5d05a-24a1-4476-894e-ba216f49f782-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:41:29.587585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.587483 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:41:29.587585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.587496 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrj2l\" (UniqueName: \"kubernetes.io/projected/32f5d05a-24a1-4476-894e-ba216f49f782-kube-api-access-nrj2l\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:41:29.587585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.587508 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f5d05a-24a1-4476-894e-ba216f49f782-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:41:29.588954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.588935 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-rkxkl"] Oct 30 04:41:29.780215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:29.780183 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:30.498671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:30.498638 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:30.846617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:30.846540 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f5d05a-24a1-4476-894e-ba216f49f782" path="/var/lib/kubelet/pods/32f5d05a-24a1-4476-894e-ba216f49f782/volumes" Oct 30 04:41:31.410631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:31.410578 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:33.072928 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:33.072893 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:34.833164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:34.833119 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:35.322566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:35.322526 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:36.077714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.077672 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd"] Oct 30 04:41:36.078608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.078583 4793 cpu_manager.go:414] "containerMap: removing container" podUID="32f5d05a-24a1-4476-894e-ba216f49f782" containerName="istio-proxy" Oct 30 04:41:36.078706 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.078610 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f5d05a-24a1-4476-894e-ba216f49f782" containerName="istio-proxy" Oct 30 04:41:36.078767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.078757 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f5d05a-24a1-4476-894e-ba216f49f782" containerName="istio-proxy" Oct 30 04:41:36.082571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.082547 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.111692 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.111667 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-4c2415-openshift-default-dockercfg-rrdsn\"" Oct 30 04:41:36.117390 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.117358 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd"] Oct 30 04:41:36.165398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.165363 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-socket\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.165582 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.165457 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlrsq\" (UniqueName: \"kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-kube-api-access-vlrsq\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.165654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.165574 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-envoy\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.165654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.165621 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-data\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.165764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.165709 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istiod-ca-cert\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.165764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.165740 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-podinfo\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.165867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.165847 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-credential-socket\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.165922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.165885 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-token\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.165922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.165914 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-certs\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.267361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.267330 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-socket\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.267534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.267411 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlrsq\" (UniqueName: \"kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-kube-api-access-vlrsq\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.267534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.267489 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-envoy\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.267534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.267520 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-data\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.267703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.267569 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istiod-ca-cert\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.267703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.267595 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-podinfo\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.267703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.267656 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-credential-socket\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.267703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.267694 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-token\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.267926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.267721 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-certs\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.267926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.267784 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-socket\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.268042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.267938 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-data\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.268109 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.268036 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-credential-socket\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.268172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.268124 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-certs\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.268726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.268701 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istiod-ca-cert\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.269779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.269752 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-envoy\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.270312 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.270288 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-podinfo\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.291350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.291327 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-token\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.291484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.291409 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlrsq\" (UniqueName: \"kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-kube-api-access-vlrsq\") pod \"e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.396002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.395909 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:36.628606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.628571 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:36.712134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.712107 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd"] Oct 30 04:41:36.712944 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:41:36.712920 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e4ad08_7b22_4083_b11a_3ca8c5ab091c.slice/crio-114d4607820150df8dc0d0b5e7fa6d5cb218303a5732f33a26f0d65aab3e3d0a WatchSource:0}: Error finding container 114d4607820150df8dc0d0b5e7fa6d5cb218303a5732f33a26f0d65aab3e3d0a: Status 404 returned error can't find the container with id 114d4607820150df8dc0d0b5e7fa6d5cb218303a5732f33a26f0d65aab3e3d0a Oct 30 04:41:36.715639 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.715605 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:41:36.715717 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.715690 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:41:36.715768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:36.715730 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:41:37.595935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:37.595899 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" event={"ID":"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c","Type":"ContainerStarted","Data":"4bc5c4cd4df53a76572b82723afc729b222cdf6cac5f4a522839ce98a279b215"} Oct 30 04:41:37.596361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:37.595960 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" event={"ID":"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c","Type":"ContainerStarted","Data":"114d4607820150df8dc0d0b5e7fa6d5cb218303a5732f33a26f0d65aab3e3d0a"} Oct 30 04:41:37.646561 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:37.646510 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" podStartSLOduration=1.646497411 podStartE2EDuration="1.646497411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:41:37.642937499 +0000 UTC m=+2620.928812286" watchObservedRunningTime="2026-10-30 04:41:37.646497411 +0000 UTC m=+2620.932372186" Oct 30 04:41:38.396303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:38.396262 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:38.398142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:38.398112 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 04:41:38.398262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:38.398172 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" podUID="e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 04:41:38.415286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:38.415252 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:39.396822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:39.396778 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 04:41:39.397304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:39.396847 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" podUID="e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 04:41:39.687205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:39.687171 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:40.093501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:40.093403 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:40.396694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:40.396588 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 04:41:40.396694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:40.396653 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" podUID="e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 04:41:41.220666 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:41:41.220634 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:41:41.396280 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:41.396247 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 04:41:41.396472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:41.396298 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" podUID="e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 04:41:41.848336 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:41.848297 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:42.397066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:42.397036 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body= Oct 30 04:41:42.397255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:42.397087 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" podUID="e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" Oct 30 04:41:43.401617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:43.401588 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:43.402072 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:43.402054 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:43.402645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:43.402626 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 04:41:43.497159 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:43.497120 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:45.182988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:45.182955 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:45.408916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:45.408881 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:45.625303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:45.625229 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:46.202286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:46.202250 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:46.415226 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:46.415188 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:46.955045 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:46.955009 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:47.178716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:47.178681 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:48.833043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:48.833001 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:49.929280 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:49.929244 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:50.164142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:50.164093 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:50.721803 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:50.721758 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:51.712748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.712714 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:51.776915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.776874 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5"] Oct 30 04:41:51.780865 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.780842 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" Oct 30 04:41:51.811496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.811466 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5"] Oct 30 04:41:51.818214 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.818191 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-bbce72\"" Oct 30 04:41:51.818367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.818250 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-bbce72-dockercfg-s97c2\"" Oct 30 04:41:51.849236 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.849207 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxtjt\" (UniqueName: \"kubernetes.io/projected/47647be1-331a-4103-86f9-fdb6b83d2d9d-kube-api-access-cxtjt\") pod \"payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5\" (UID: \"47647be1-331a-4103-86f9-fdb6b83d2d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" Oct 30 04:41:51.849348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.849324 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47647be1-331a-4103-86f9-fdb6b83d2d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5\" (UID: \"47647be1-331a-4103-86f9-fdb6b83d2d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" Oct 30 04:41:51.863580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.863548 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f"] Oct 30 04:41:51.866728 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.866710 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" Oct 30 04:41:51.936437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.935470 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f"] Oct 30 04:41:51.956606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.954777 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxtjt\" (UniqueName: \"kubernetes.io/projected/47647be1-331a-4103-86f9-fdb6b83d2d9d-kube-api-access-cxtjt\") pod \"payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5\" (UID: \"47647be1-331a-4103-86f9-fdb6b83d2d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" Oct 30 04:41:51.956606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.954946 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47647be1-331a-4103-86f9-fdb6b83d2d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5\" (UID: \"47647be1-331a-4103-86f9-fdb6b83d2d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" Oct 30 04:41:51.956606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.954978 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f\" (UID: \"43ca6af4-392d-4c3a-b8de-9cbb133ff81d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" Oct 30 04:41:51.956606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.955070 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77xw4\" (UniqueName: \"kubernetes.io/projected/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-kube-api-access-77xw4\") pod \"payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f\" (UID: \"43ca6af4-392d-4c3a-b8de-9cbb133ff81d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" Oct 30 04:41:51.956606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.956082 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47647be1-331a-4103-86f9-fdb6b83d2d9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5\" (UID: \"47647be1-331a-4103-86f9-fdb6b83d2d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" Oct 30 04:41:51.990232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.990156 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxtjt\" (UniqueName: \"kubernetes.io/projected/47647be1-331a-4103-86f9-fdb6b83d2d9d-kube-api-access-cxtjt\") pod \"payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5\" (UID: \"47647be1-331a-4103-86f9-fdb6b83d2d9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" Oct 30 04:41:51.990816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:51.990788 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:52.056336 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.056307 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77xw4\" (UniqueName: \"kubernetes.io/projected/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-kube-api-access-77xw4\") pod \"payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f\" (UID: \"43ca6af4-392d-4c3a-b8de-9cbb133ff81d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" Oct 30 04:41:52.056510 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.056495 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f\" (UID: \"43ca6af4-392d-4c3a-b8de-9cbb133ff81d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" Oct 30 04:41:52.057145 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.057126 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f\" (UID: \"43ca6af4-392d-4c3a-b8de-9cbb133ff81d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" Oct 30 04:41:52.082188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.082163 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77xw4\" (UniqueName: \"kubernetes.io/projected/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-kube-api-access-77xw4\") pod \"payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f\" (UID: \"43ca6af4-392d-4c3a-b8de-9cbb133ff81d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" Oct 30 04:41:52.095688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.095658 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" Oct 30 04:41:52.262344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.262259 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" Oct 30 04:41:52.411975 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:41:52.411897 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47647be1_331a_4103_86f9_fdb6b83d2d9d.slice/crio-d003cbf62222e66743c3ca49999de5bbab9648c98b5e87eba9572919391b9024 WatchSource:0}: Error finding container d003cbf62222e66743c3ca49999de5bbab9648c98b5e87eba9572919391b9024: Status 404 returned error can't find the container with id d003cbf62222e66743c3ca49999de5bbab9648c98b5e87eba9572919391b9024 Oct 30 04:41:52.413042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.413010 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5"] Oct 30 04:41:52.574862 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.574836 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f"] Oct 30 04:41:52.574984 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:41:52.574833 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ca6af4_392d_4c3a_b8de_9cbb133ff81d.slice/crio-cdf931d036bb616e59bfc4a582634c03ea7cb0b2740a5cf9f3456273650dfae6 WatchSource:0}: Error finding container cdf931d036bb616e59bfc4a582634c03ea7cb0b2740a5cf9f3456273650dfae6: Status 404 returned error can't find the container with id cdf931d036bb616e59bfc4a582634c03ea7cb0b2740a5cf9f3456273650dfae6 Oct 30 04:41:52.603806 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.603777 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:52.705818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.705781 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" event={"ID":"43ca6af4-392d-4c3a-b8de-9cbb133ff81d","Type":"ContainerStarted","Data":"58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d"} Oct 30 04:41:52.706040 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.705843 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" Oct 30 04:41:52.706040 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.705865 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" event={"ID":"43ca6af4-392d-4c3a-b8de-9cbb133ff81d","Type":"ContainerStarted","Data":"cdf931d036bb616e59bfc4a582634c03ea7cb0b2740a5cf9f3456273650dfae6"} Oct 30 04:41:52.707723 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.707695 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" event={"ID":"47647be1-331a-4103-86f9-fdb6b83d2d9d","Type":"ContainerStarted","Data":"4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda"} Oct 30 04:41:52.707833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.707753 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" Oct 30 04:41:52.707833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.707776 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" event={"ID":"47647be1-331a-4103-86f9-fdb6b83d2d9d","Type":"ContainerStarted","Data":"d003cbf62222e66743c3ca49999de5bbab9648c98b5e87eba9572919391b9024"} Oct 30 04:41:52.748870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.748819 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" podStartSLOduration=1.748806966 podStartE2EDuration="1.748806966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:41:52.745907953 +0000 UTC m=+2636.031782727" watchObservedRunningTime="2026-10-30 04:41:52.748806966 +0000 UTC m=+2636.034681740" Oct 30 04:41:52.788471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:52.788407 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" podStartSLOduration=1.7883923720000001 podStartE2EDuration="1.788392372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:41:52.786558661 +0000 UTC m=+2636.072433441" watchObservedRunningTime="2026-10-30 04:41:52.788392372 +0000 UTC m=+2636.074267147" Oct 30 04:41:54.374621 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:54.374585 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:55.226103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:55.226064 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:55.981279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:55.981227 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:57.616983 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:57.616933 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:41:58.580002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:58.579971 4793 scope.go:122] "RemoveContainer" containerID="47f7cc3e6bfa0f014370406cee1dbcc9ac15a4c30243bc1e8ac800f6216a2e48" Oct 30 04:41:58.589760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:58.589740 4793 scope.go:122] "RemoveContainer" containerID="90e6d62cef63ac368ba7b4b091007914490ae85e9fc8f9ef2d939fe1197f8908" Oct 30 04:41:58.598366 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:58.598331 4793 scope.go:122] "RemoveContainer" containerID="2c4c29ec4dba66a9213281e83d7d23b79555d56deb932495fb2e1647604ec05b" Oct 30 04:41:59.434881 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:41:59.434843 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:01.112681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:01.112643 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:02.712625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:02.712592 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:03.718385 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:03.718356 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" Oct 30 04:42:03.718793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:03.718442 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" Oct 30 04:42:04.216099 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:04.216061 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:04.503097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:04.503008 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:06.210873 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:06.210842 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:07.817053 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:07.817019 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:09.490308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:09.490119 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:10.452138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.452103 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd"] Oct 30 04:42:10.458284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.458259 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.481189 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.481162 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd"] Oct 30 04:42:10.487733 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.487709 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-495ebe-openshift-default-dockercfg-76kt2\"" Oct 30 04:42:10.595143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.595108 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-envoy\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.595588 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.595227 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb7e2e-f048-4520-93e1-7438a80fba50-istiod-ca-cert\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.595588 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.595270 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-credential-socket\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.595588 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.595451 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-socket\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.595588 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.595484 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-podinfo\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.595588 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.595504 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-token\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.595815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.595620 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-data\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.595815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.595659 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-certs\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.595815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.595678 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9hl2\" (UniqueName: \"kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-kube-api-access-d9hl2\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.697593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.697558 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-data\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.697765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.697633 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-certs\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.697765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.697663 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9hl2\" (UniqueName: \"kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-kube-api-access-d9hl2\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.697765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.697743 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-envoy\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.697967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.697806 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb7e2e-f048-4520-93e1-7438a80fba50-istiod-ca-cert\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.697967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.697844 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-credential-socket\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.697967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.697940 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-data\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.698128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.697973 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-socket\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.698128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.697986 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-certs\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.698128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.698003 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-podinfo\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.698128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.698043 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-token\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.698346 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.698310 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-credential-socket\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.698471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.698449 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-socket\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.698801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.698772 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb7e2e-f048-4520-93e1-7438a80fba50-istiod-ca-cert\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.699975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.699953 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-envoy\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.700534 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.700517 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-podinfo\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.721201 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.721154 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-token\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.721330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.721289 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9hl2\" (UniqueName: \"kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-kube-api-access-d9hl2\") pod \"e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:10.774114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:10.774089 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:11.098482 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:42:11.098448 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ceb7e2e_f048_4520_93e1_7438a80fba50.slice/crio-81874562f8dddb315cd80bb79ca79eb92b56641f5f5b73c2f504f7f1ba0eb9df WatchSource:0}: Error finding container 81874562f8dddb315cd80bb79ca79eb92b56641f5f5b73c2f504f7f1ba0eb9df: Status 404 returned error can't find the container with id 81874562f8dddb315cd80bb79ca79eb92b56641f5f5b73c2f504f7f1ba0eb9df Oct 30 04:42:11.098634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:11.098601 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd"] Oct 30 04:42:11.101692 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:11.101646 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:42:11.101800 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:11.101738 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:42:11.101800 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:11.101783 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:42:11.223538 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:42:11.223497 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:42:11.319305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:11.319271 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:11.858114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:11.858080 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" event={"ID":"3ceb7e2e-f048-4520-93e1-7438a80fba50","Type":"ContainerStarted","Data":"ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19"} Oct 30 04:42:11.858663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:11.858131 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" event={"ID":"3ceb7e2e-f048-4520-93e1-7438a80fba50","Type":"ContainerStarted","Data":"81874562f8dddb315cd80bb79ca79eb92b56641f5f5b73c2f504f7f1ba0eb9df"} Oct 30 04:42:12.787444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:12.776984 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:12.787444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:12.780525 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 30 04:42:12.787444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:12.780584 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" podUID="3ceb7e2e-f048-4520-93e1-7438a80fba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 30 04:42:12.986889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:12.986836 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:13.209582 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:13.209537 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:13.779827 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:13.779784 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 30 04:42:13.780125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:13.780094 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" podUID="3ceb7e2e-f048-4520-93e1-7438a80fba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 30 04:42:14.778442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:14.774881 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 30 04:42:14.778442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:14.774962 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" podUID="3ceb7e2e-f048-4520-93e1-7438a80fba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 30 04:42:14.844668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:14.844625 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:15.167510 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:15.166890 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:15.784476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:15.781921 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 30 04:42:15.784476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:15.781996 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" podUID="3ceb7e2e-f048-4520-93e1-7438a80fba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 30 04:42:16.593815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:16.593779 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:16.778895 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:16.778856 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Oct 30 04:42:16.779112 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:16.778914 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" podUID="3ceb7e2e-f048-4520-93e1-7438a80fba50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Oct 30 04:42:17.247403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:17.247372 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:18.049288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:18.049164 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:18.050364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:18.050340 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:18.050971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:18.050933 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 04:42:18.116897 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:18.116847 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" podStartSLOduration=8.116833149 podStartE2EDuration="8.116833149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:42:11.9188079 +0000 UTC m=+2655.204682676" watchObservedRunningTime="2026-10-30 04:42:18.116833149 +0000 UTC m=+2661.402707923" Oct 30 04:42:18.533182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:18.533151 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:20.131526 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:20.131492 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:21.828687 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:21.828649 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:23.526463 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:23.526410 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:24.521554 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:24.521516 4793 patch_prober.go:29] interesting pod/e2e-explicit-20a650-openshift-default-54678c495c-j52w6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:42:24.521756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:24.521591 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:42:25.207854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:25.207822 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:26.026842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:26.026810 4793 generic.go:356] "Generic (PLEG): container finished" podID="138e743d-b5cc-4304-a424-e52af2ffd818" containerID="3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e" exitCode=0 Oct 30 04:42:26.027073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:26.026885 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" event={"ID":"138e743d-b5cc-4304-a424-e52af2ffd818","Type":"ContainerDied","Data":"3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e"} Oct 30 04:42:26.027352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:26.027333 4793 scope.go:122] "RemoveContainer" containerID="3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e" Oct 30 04:42:26.028962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:26.028928 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:42:26.029058 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:26.029018 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:42:26.029098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:26.029064 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:42:26.886721 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:26.886688 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:27.036591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:27.036559 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" event={"ID":"138e743d-b5cc-4304-a424-e52af2ffd818","Type":"ContainerStarted","Data":"a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe"} Oct 30 04:42:27.946686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:27.946643 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:42:27.947181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:27.946883 4793 patch_prober.go:29] interesting pod/e2e-explicit-20a650-openshift-default-54678c495c-j52w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 04:42:27.947181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:27.946949 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 04:42:28.606740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:28.606708 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:28.948006 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:28.947963 4793 patch_prober.go:29] interesting pod/e2e-explicit-20a650-openshift-default-54678c495c-j52w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 04:42:28.948487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:28.948044 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 04:42:29.947371 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:29.947333 4793 patch_prober.go:29] interesting pod/e2e-explicit-20a650-openshift-default-54678c495c-j52w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 04:42:29.947587 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:29.947390 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 04:42:30.217962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:30.217885 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:30.789939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:30.789852 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:30.947090 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:30.947048 4793 patch_prober.go:29] interesting pod/e2e-explicit-20a650-openshift-default-54678c495c-j52w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 04:42:30.947283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:30.947108 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 04:42:31.245511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:31.245481 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:31.812154 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:31.812112 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:31.947615 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:31.947577 4793 patch_prober.go:29] interesting pod/e2e-explicit-20a650-openshift-default-54678c495c-j52w6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" start-of-body= Oct 30 04:42:31.947794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:31.947633 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.188:15021/healthz/ready\": dial tcp 10.133.0.188:15021: connect: connection refused" Oct 30 04:42:32.951573 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:32.951545 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:42:32.952032 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:32.951978 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:42:32.952645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:32.952625 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:42:33.606854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:33.606815 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:35.217552 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:35.217519 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:36.374853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:36.374812 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:36.818841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:36.818804 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:38.496803 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:38.496762 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:40.216091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:40.216053 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:41.219663 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:42:41.219628 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:42:42.112516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:42.112445 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:43.821821 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:43.821779 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:45.507268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:45.507229 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:45.737631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:45.737594 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:47.253682 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:47.253640 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:48.840960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:48.840923 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:50.524731 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:50.524680 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:50.949034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:50.949000 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:52.185523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:52.185490 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:54.024086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:54.024041 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:55.335140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:55.335104 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:55.578398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:55.578360 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:55.832610 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:55.832578 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:57.387623 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:57.387579 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:42:59.038968 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:42:59.038936 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:00.725399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:00.725359 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:02.418107 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:02.418064 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:04.218402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:04.218363 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:05.959794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:05.959756 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:07.820671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:07.820634 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:09.031898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:09.031868 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw"] Oct 30 04:43:09.043364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:09.043338 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d8bdbb3-4vbcw"] Oct 30 04:43:09.561593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:09.561550 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:10.847131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:10.847099 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0" path="/var/lib/kubelet/pods/8ef92d91-6d4f-42a6-ad9a-2d2bc6c481f0/volumes" Oct 30 04:43:11.127721 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:11.127636 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:11.220841 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:43:11.220805 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:43:12.738515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:12.738475 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:13.248095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:13.248058 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:13.466342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:13.466310 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:14.144471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:14.144432 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:14.508629 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:14.508595 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:14.711654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:14.711621 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:16.185843 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:16.185805 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:17.908916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:17.908879 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:19.799759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:19.799726 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:20.524744 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:20.524708 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:21.449243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:21.449212 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:22.111061 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:22.111025 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:23.081497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:23.081461 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:24.754733 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:24.754692 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:26.450992 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:26.450956 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:28.024511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:28.024477 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:28.607350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:28.607308 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:30.290253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:30.290213 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:31.979580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:31.979541 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:33.600808 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:33.600769 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:35.215233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:35.215202 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:35.424330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:35.424266 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:35.933962 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:35.933911 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:36.333124 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:36.333052 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6"] Oct 30 04:43:36.334210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:36.334167 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" containerID="cri-o://a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe" gracePeriod=30 Oct 30 04:43:36.530067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:36.529988 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:36.745496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:36.745458 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:37.163089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:37.162987 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:37.385124 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:37.385088 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:37.611866 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:37.611831 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:37.813636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:37.813595 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:38.974050 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:38.974016 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:39.533007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:39.532932 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:39.740540 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:39.740498 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:39.974232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:39.974196 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:40.859926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:40.859885 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:41.219975 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:43:41.219942 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:43:41.618915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.618892 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:43:41.661886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.661856 4793 generic.go:356] "Generic (PLEG): container finished" podID="138e743d-b5cc-4304-a424-e52af2ffd818" containerID="a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe" exitCode=0 Oct 30 04:43:41.662049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.661894 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" event={"ID":"138e743d-b5cc-4304-a424-e52af2ffd818","Type":"ContainerDied","Data":"a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe"} Oct 30 04:43:41.662049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.661900 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" Oct 30 04:43:41.662049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.661945 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6" event={"ID":"138e743d-b5cc-4304-a424-e52af2ffd818","Type":"ContainerDied","Data":"4205161c0429b43e5b2910d422547770f6cf1b673faeb31f5df86dab0891a34c"} Oct 30 04:43:41.662049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.661970 4793 scope.go:122] "RemoveContainer" containerID="a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe" Oct 30 04:43:41.672285 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.672264 4793 scope.go:122] "RemoveContainer" containerID="3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e" Oct 30 04:43:41.682745 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.682728 4793 scope.go:122] "RemoveContainer" containerID="a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe" Oct 30 04:43:41.683026 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:43:41.683003 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe\": container with ID starting with a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe not found: ID does not exist" containerID="a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe" Oct 30 04:43:41.683130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683031 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe"} err="failed to get container status \"a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe\": rpc error: code = NotFound desc = could not find container \"a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe\": container with ID starting with a2b4475a4dfe8eab9d163f3ee05856cdc8cc6ad810a62643496e3ef6502ea4fe not found: ID does not exist" Oct 30 04:43:41.683130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683056 4793 scope.go:122] "RemoveContainer" containerID="3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e" Oct 30 04:43:41.683130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683083 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-credential-socket\") pod \"138e743d-b5cc-4304-a424-e52af2ffd818\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " Oct 30 04:43:41.683130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683121 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-kube-api-access-gnfb5\" (UniqueName: \"kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-kube-api-access-gnfb5\") pod \"138e743d-b5cc-4304-a424-e52af2ffd818\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " Oct 30 04:43:41.683375 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683206 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-data\") pod \"138e743d-b5cc-4304-a424-e52af2ffd818\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " Oct 30 04:43:41.683375 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683280 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-istio-token\" (UniqueName: \"kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-istio-token\") pod \"138e743d-b5cc-4304-a424-e52af2ffd818\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " Oct 30 04:43:41.683375 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683310 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-socket\") pod \"138e743d-b5cc-4304-a424-e52af2ffd818\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " Oct 30 04:43:41.683375 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683346 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-credential-socket" pod "138e743d-b5cc-4304-a424-e52af2ffd818" (UID: "138e743d-b5cc-4304-a424-e52af2ffd818"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:43:41.683617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683389 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-envoy\") pod \"138e743d-b5cc-4304-a424-e52af2ffd818\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " Oct 30 04:43:41.683617 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:43:41.683459 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e\": container with ID starting with 3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e not found: ID does not exist" containerID="3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e" Oct 30 04:43:41.683617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683482 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/138e743d-b5cc-4304-a424-e52af2ffd818-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138e743d-b5cc-4304-a424-e52af2ffd818-istio-podinfo\") pod \"138e743d-b5cc-4304-a424-e52af2ffd818\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " Oct 30 04:43:41.683617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683490 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e"} err="failed to get container status \"3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e\": rpc error: code = NotFound desc = could not find container \"3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e\": container with ID starting with 3aa0c8ab13eb20a8dfccde7e800be1a6ce52bc4f01fc344a97d5e9511239124e not found: ID does not exist" Oct 30 04:43:41.683617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683511 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138e743d-b5cc-4304-a424-e52af2ffd818-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138e743d-b5cc-4304-a424-e52af2ffd818-istiod-ca-cert\") pod \"138e743d-b5cc-4304-a424-e52af2ffd818\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " Oct 30 04:43:41.683617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683583 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-certs\") pod \"138e743d-b5cc-4304-a424-e52af2ffd818\" (UID: \"138e743d-b5cc-4304-a424-e52af2ffd818\") " Oct 30 04:43:41.683947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.683798 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-socket" pod "138e743d-b5cc-4304-a424-e52af2ffd818" (UID: "138e743d-b5cc-4304-a424-e52af2ffd818"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:43:41.684288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.684106 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-data" pod "138e743d-b5cc-4304-a424-e52af2ffd818" (UID: "138e743d-b5cc-4304-a424-e52af2ffd818"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:43:41.684288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.684170 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-certs" pod "138e743d-b5cc-4304-a424-e52af2ffd818" (UID: "138e743d-b5cc-4304-a424-e52af2ffd818"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:43:41.684631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.684591 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138e743d-b5cc-4304-a424-e52af2ffd818-istiod-ca-cert" pod "138e743d-b5cc-4304-a424-e52af2ffd818" (UID: "138e743d-b5cc-4304-a424-e52af2ffd818"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:43:41.684994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.684923 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:43:41.684994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.684946 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:43:41.684994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.684957 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:43:41.684994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.684969 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138e743d-b5cc-4304-a424-e52af2ffd818-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:43:41.684994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.684977 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:43:41.686448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.686388 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-kube-api-access-gnfb5" pod "138e743d-b5cc-4304-a424-e52af2ffd818" (UID: "138e743d-b5cc-4304-a424-e52af2ffd818"). InnerVolumeSpecName "kube-api-access-gnfb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:43:41.686670 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.686640 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-envoy" pod "138e743d-b5cc-4304-a424-e52af2ffd818" (UID: "138e743d-b5cc-4304-a424-e52af2ffd818"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:43:41.687996 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.687974 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-istio-token" pod "138e743d-b5cc-4304-a424-e52af2ffd818" (UID: "138e743d-b5cc-4304-a424-e52af2ffd818"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:43:41.688064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.688006 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/138e743d-b5cc-4304-a424-e52af2ffd818-istio-podinfo" pod "138e743d-b5cc-4304-a424-e52af2ffd818" (UID: "138e743d-b5cc-4304-a424-e52af2ffd818"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:43:41.786233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.786166 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:43:41.786233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.786191 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138e743d-b5cc-4304-a424-e52af2ffd818-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:43:41.786233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.786202 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138e743d-b5cc-4304-a424-e52af2ffd818-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:43:41.786233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:41.786211 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnfb5\" (UniqueName: \"kubernetes.io/projected/138e743d-b5cc-4304-a424-e52af2ffd818-kube-api-access-gnfb5\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:43:42.014633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:42.014599 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6"] Oct 30 04:43:42.026884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:42.026857 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-20a650-openshift-default-54678c495c-j52w6"] Oct 30 04:43:42.513538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:42.513500 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:42.847564 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:42.847492 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" path="/var/lib/kubelet/pods/138e743d-b5cc-4304-a424-e52af2ffd818/volumes" Oct 30 04:43:44.204283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:44.204244 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:44.436567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:44.436531 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:45.944034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:45.944001 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:47.708134 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:47.708100 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:49.407864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:49.407807 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:51.107391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:51.107347 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:52.789827 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:52.789783 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:54.529367 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:54.529324 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:55.475898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:55.475859 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:56.309437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:56.309392 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:56.530757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:56.530725 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:56.737565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:56.737517 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:56.945870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:56.945829 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:57.153851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:57.153770 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:58.101483 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:58.101438 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:43:59.913920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:43:59.913885 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:00.750142 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:00.750100 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:00.962633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:00.962592 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:01.704277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:01.704218 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:02.434831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:02.434796 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:02.648287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:02.648252 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:03.479033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:03.478995 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:05.090120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:05.090068 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:05.313626 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:05.313589 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:05.609668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:05.609633 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:05.815819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:05.815784 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:06.943649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:06.943609 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:08.591657 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:08.591616 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:10.282926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:10.282881 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:10.806198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:10.806165 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:11.219348 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:44:11.219311 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:44:11.880125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:11.880085 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:13.606257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:13.606217 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:15.277186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:15.277144 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:16.983896 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:16.983844 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:18.710318 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:18.710279 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:20.522841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:20.522801 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:22.245321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:22.245281 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:23.864259 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:23.864222 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:24.085266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:24.085233 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:24.288515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:24.288479 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:25.613195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:25.613162 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:27.334702 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:27.334669 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:29.002788 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:29.002748 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:30.802523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:30.802481 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:32.508347 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:32.508303 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:34.179646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:34.179607 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:35.787795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:35.787754 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:37.381805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:37.381767 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:38.983192 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:38.983154 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:40.573461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:40.573401 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:41.219733 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:44:41.219694 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:44:42.382544 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:42.382503 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:42.588543 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:42.588488 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:44.087274 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:44.087192 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:45.689875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:45.689839 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:47.420618 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:47.420575 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:48.122753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:48.122721 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:49.202498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:49.202446 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:50.441115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:50.441075 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:50.970880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:50.970846 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:51.447721 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:51.447689 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:51.786982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:51.786895 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:52.118125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.118042 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml"] Oct 30 04:44:52.118987 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.118966 4793 cpu_manager.go:414] "containerMap: removing container" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" Oct 30 04:44:52.119061 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.118990 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" Oct 30 04:44:52.119215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.119202 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" Oct 30 04:44:52.119255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.119228 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" Oct 30 04:44:52.123554 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.123518 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.161658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.161630 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml"] Oct 30 04:44:52.197465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.193245 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-certs\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.197465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.193340 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-data\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.197465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.193492 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-istio-token\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.197465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.193610 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07de9748-a77d-4181-8202-6edb393a4882-istiod-ca-cert\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.197465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.193644 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-socket\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.197465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.193710 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-credential-socket\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.197465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.193762 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07de9748-a77d-4181-8202-6edb393a4882-istio-podinfo\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.197465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.193825 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glcs2\" (UniqueName: \"kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-kube-api-access-glcs2\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.197465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.193930 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-envoy\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.219506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.219480 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6378cc-openshift-default-dockercfg-cwxgg\"" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.299234 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-certs\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.299302 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-data\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.299339 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-istio-token\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.299399 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07de9748-a77d-4181-8202-6edb393a4882-istiod-ca-cert\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.299462 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-socket\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.299529 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-credential-socket\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.299580 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07de9748-a77d-4181-8202-6edb393a4882-istio-podinfo\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.299639 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glcs2\" (UniqueName: \"kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-kube-api-access-glcs2\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.299748 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-envoy\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.300684 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-data\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.300906 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-certs\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.301112 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-credential-socket\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.301220 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07de9748-a77d-4181-8202-6edb393a4882-istiod-ca-cert\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.303440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.301315 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-socket\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.306305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.306145 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:52.313525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.313498 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07de9748-a77d-4181-8202-6edb393a4882-istio-podinfo\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.315559 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.315528 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-envoy\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.331993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.331963 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-istio-token\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.332892 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.332868 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glcs2\" (UniqueName: \"kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-kube-api-access-glcs2\") pod \"e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.442233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.442201 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:52.518182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.518131 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:52.785794 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:44:52.785760 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07de9748_a77d_4181_8202_6edb393a4882.slice/crio-d5c77dc7bc6c5a266c1d249405e7fe396e84216108d79deb08bf70585b83c413 WatchSource:0}: Error finding container d5c77dc7bc6c5a266c1d249405e7fe396e84216108d79deb08bf70585b83c413: Status 404 returned error can't find the container with id d5c77dc7bc6c5a266c1d249405e7fe396e84216108d79deb08bf70585b83c413 Oct 30 04:44:52.788461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.788402 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:44:52.788566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.788507 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:44:52.788566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.788540 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:44:52.788645 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.788588 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml"] Oct 30 04:44:52.866308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:52.866270 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:53.079271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.079182 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl"] Oct 30 04:44:53.080460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.080430 4793 cpu_manager.go:414] "containerMap: removing container" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" Oct 30 04:44:53.080460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.080458 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138e743d-b5cc-4304-a424-e52af2ffd818" containerName="istio-proxy" Oct 30 04:44:53.084964 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.084940 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.114232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.112771 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:53.122402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.122372 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl"] Oct 30 04:44:53.123927 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.123908 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-483bea-openshift-default-dockercfg-xdxcd\"" Oct 30 04:44:53.206000 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.205961 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" event={"ID":"07de9748-a77d-4181-8202-6edb393a4882","Type":"ContainerStarted","Data":"ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba"} Oct 30 04:44:53.206181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.206029 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" event={"ID":"07de9748-a77d-4181-8202-6edb393a4882","Type":"ContainerStarted","Data":"d5c77dc7bc6c5a266c1d249405e7fe396e84216108d79deb08bf70585b83c413"} Oct 30 04:44:53.212976 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.212947 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-envoy\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.213236 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.213005 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-credential-socket\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.213236 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.213107 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-certs\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.213236 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.213145 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-token\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.213411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.213262 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istiod-ca-cert\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.213411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.213308 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9w8v\" (UniqueName: \"kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-kube-api-access-x9w8v\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.213411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.213392 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-socket\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.213548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.213440 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-data\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.213548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.213457 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-podinfo\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.263045 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.262995 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" podStartSLOduration=1.262982008 podStartE2EDuration="1.262982008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:53.26152573 +0000 UTC m=+2816.547400509" watchObservedRunningTime="2026-10-30 04:44:53.262982008 +0000 UTC m=+2816.548856783" Oct 30 04:44:53.308556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.308513 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:53.314933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.314902 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-socket\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.315021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.314965 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-data\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.315021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.314991 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-podinfo\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.315131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.315108 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-envoy\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.315190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.315178 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-credential-socket\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.315269 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.315253 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-certs\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.315343 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.315313 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-token\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.315399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.315365 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istiod-ca-cert\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.315517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.315457 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9w8v\" (UniqueName: \"kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-kube-api-access-x9w8v\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.315618 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.315594 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-data\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.315686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.315635 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-credential-socket\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.315914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.315888 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-certs\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.316454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.316411 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istiod-ca-cert\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.316576 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.316528 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-socket\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.318147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.318121 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-envoy\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.318301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.318277 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-podinfo\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.339098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.339044 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-token\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.339319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.339300 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9w8v\" (UniqueName: \"kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-kube-api-access-x9w8v\") pod \"e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.432442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.430600 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:53.453853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.453745 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:53.530593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.521605 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:53.536411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.536374 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:53.842789 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:44:53.842753 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9fbe4b_5c65_4e1c_a3f7_6a174131fd2e.slice/crio-86be27ef33979621c1e4490130757fc2d931b9e7daabda5f482d53d28b85e4bb WatchSource:0}: Error finding container 86be27ef33979621c1e4490130757fc2d931b9e7daabda5f482d53d28b85e4bb: Status 404 returned error can't find the container with id 86be27ef33979621c1e4490130757fc2d931b9e7daabda5f482d53d28b85e4bb Oct 30 04:44:53.843076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.843056 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl"] Oct 30 04:44:53.845536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.845508 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:44:53.845631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.845568 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:44:53.845631 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:53.845596 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:44:54.215349 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:54.215311 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" event={"ID":"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e","Type":"ContainerStarted","Data":"026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56"} Oct 30 04:44:54.215538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:54.215374 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" event={"ID":"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e","Type":"ContainerStarted","Data":"86be27ef33979621c1e4490130757fc2d931b9e7daabda5f482d53d28b85e4bb"} Oct 30 04:44:54.215696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:54.215623 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:54.216592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:54.216567 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:44:54.262895 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:54.262838 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" podStartSLOduration=1.262821021 podStartE2EDuration="1.262821021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:44:54.261380965 +0000 UTC m=+2817.547255745" watchObservedRunningTime="2026-10-30 04:44:54.262821021 +0000 UTC m=+2817.548695797" Oct 30 04:44:54.443012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:54.442789 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:54.457189 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:54.457159 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:54.625327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:54.625234 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:55.230342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:55.226207 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:55.230342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:55.228155 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:44:55.376292 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:55.376254 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:55.619722 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:55.619643 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:55.931464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:55.931401 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:56.557218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:56.557180 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:56.844254 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:56.844147 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:57.337188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:57.337141 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:57.576991 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:57.576957 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:58.524403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:58.524359 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:58.664938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:58.664910 4793 scope.go:122] "RemoveContainer" containerID="460618eab1b0b517128d5a6377e439d16147d7dea3c2f9c20a26dd6bda61da5c" Oct 30 04:44:59.023850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:59.023815 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:59.311336 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:59.309560 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:44:59.758478 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:44:59.758409 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:00.225434 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.225393 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5"] Oct 30 04:45:00.231144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.231119 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" Oct 30 04:45:00.299641 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.299612 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5"] Oct 30 04:45:00.311082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.311050 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq"] Oct 30 04:45:00.315258 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.315234 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" Oct 30 04:45:00.338490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.338469 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm"] Oct 30 04:45:00.345251 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.345228 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 04:45:00.352944 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.352919 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" Oct 30 04:45:00.365605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.365575 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5\" (UID: \"2c3d37d2-eb6c-4dd1-a80a-bceb39880640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" Oct 30 04:45:00.366021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.365981 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blpw6\" (UniqueName: \"kubernetes.io/projected/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-kube-api-access-blpw6\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5\" (UID: \"2c3d37d2-eb6c-4dd1-a80a-bceb39880640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" Oct 30 04:45:00.370124 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.369744 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq"] Oct 30 04:45:00.400055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.400022 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc"] Oct 30 04:45:00.412852 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.412826 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm"] Oct 30 04:45:00.413004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.412861 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq"] Oct 30 04:45:00.413004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.412950 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" Oct 30 04:45:00.417963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.417937 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw"] Oct 30 04:45:00.418091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.418053 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" Oct 30 04:45:00.424310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.424283 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc"] Oct 30 04:45:00.424446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.424326 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq"] Oct 30 04:45:00.424446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.424393 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" Oct 30 04:45:00.426293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.426277 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw"] Oct 30 04:45:00.467554 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.467522 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18bf49cc-fe19-48d9-8947-166ebcce843d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm\" (UID: \"18bf49cc-fe19-48d9-8947-166ebcce843d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" Oct 30 04:45:00.467671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.467598 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n5fx\" (UniqueName: \"kubernetes.io/projected/18bf49cc-fe19-48d9-8947-166ebcce843d-kube-api-access-5n5fx\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm\" (UID: \"18bf49cc-fe19-48d9-8947-166ebcce843d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" Oct 30 04:45:00.467671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.467657 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5\" (UID: \"2c3d37d2-eb6c-4dd1-a80a-bceb39880640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" Oct 30 04:45:00.467820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.467795 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac757cc2-5436-41e3-b954-d3c0230757f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq\" (UID: \"ac757cc2-5436-41e3-b954-d3c0230757f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" Oct 30 04:45:00.467886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.467853 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blpw6\" (UniqueName: \"kubernetes.io/projected/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-kube-api-access-blpw6\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5\" (UID: \"2c3d37d2-eb6c-4dd1-a80a-bceb39880640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" Oct 30 04:45:00.467950 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.467887 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb84g\" (UniqueName: \"kubernetes.io/projected/ac757cc2-5436-41e3-b954-d3c0230757f9-kube-api-access-vb84g\") pod \"maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq\" (UID: \"ac757cc2-5436-41e3-b954-d3c0230757f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" Oct 30 04:45:00.468364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.468344 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5\" (UID: \"2c3d37d2-eb6c-4dd1-a80a-bceb39880640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" Oct 30 04:45:00.496340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.496286 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blpw6\" (UniqueName: \"kubernetes.io/projected/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-kube-api-access-blpw6\") pod \"maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5\" (UID: \"2c3d37d2-eb6c-4dd1-a80a-bceb39880640\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" Oct 30 04:45:00.545283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.545245 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" Oct 30 04:45:00.569969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.569936 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1fbc60-30e0-4846-b526-f52a9c8a4091-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc\" (UID: \"2c1fbc60-30e0-4846-b526-f52a9c8a4091\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" Oct 30 04:45:00.570120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.570005 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmcxs\" (UniqueName: \"kubernetes.io/projected/2c1fbc60-30e0-4846-b526-f52a9c8a4091-kube-api-access-vmcxs\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc\" (UID: \"2c1fbc60-30e0-4846-b526-f52a9c8a4091\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" Oct 30 04:45:00.570198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.570149 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5t9\" (UniqueName: \"kubernetes.io/projected/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-kube-api-access-fd5t9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw\" (UID: \"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" Oct 30 04:45:00.570262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.570204 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq98r\" (UniqueName: \"kubernetes.io/projected/d692e5e1-80df-47e2-9dc5-64adf498e349-kube-api-access-dq98r\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq\" (UID: \"d692e5e1-80df-47e2-9dc5-64adf498e349\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" Oct 30 04:45:00.570342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.570318 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d692e5e1-80df-47e2-9dc5-64adf498e349-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq\" (UID: \"d692e5e1-80df-47e2-9dc5-64adf498e349\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" Oct 30 04:45:00.570402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.570375 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18bf49cc-fe19-48d9-8947-166ebcce843d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm\" (UID: \"18bf49cc-fe19-48d9-8947-166ebcce843d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" Oct 30 04:45:00.570511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.570487 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n5fx\" (UniqueName: \"kubernetes.io/projected/18bf49cc-fe19-48d9-8947-166ebcce843d-kube-api-access-5n5fx\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm\" (UID: \"18bf49cc-fe19-48d9-8947-166ebcce843d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" Oct 30 04:45:00.570590 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.570572 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw\" (UID: \"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" Oct 30 04:45:00.570682 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.570662 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac757cc2-5436-41e3-b954-d3c0230757f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq\" (UID: \"ac757cc2-5436-41e3-b954-d3c0230757f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" Oct 30 04:45:00.570743 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.570715 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb84g\" (UniqueName: \"kubernetes.io/projected/ac757cc2-5436-41e3-b954-d3c0230757f9-kube-api-access-vb84g\") pod \"maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq\" (UID: \"ac757cc2-5436-41e3-b954-d3c0230757f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" Oct 30 04:45:00.571315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.571285 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18bf49cc-fe19-48d9-8947-166ebcce843d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm\" (UID: \"18bf49cc-fe19-48d9-8947-166ebcce843d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" Oct 30 04:45:00.571452 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.571339 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac757cc2-5436-41e3-b954-d3c0230757f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq\" (UID: \"ac757cc2-5436-41e3-b954-d3c0230757f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" Oct 30 04:45:00.589770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.589734 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:00.595012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.594985 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb84g\" (UniqueName: \"kubernetes.io/projected/ac757cc2-5436-41e3-b954-d3c0230757f9-kube-api-access-vb84g\") pod \"maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq\" (UID: \"ac757cc2-5436-41e3-b954-d3c0230757f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" Oct 30 04:45:00.595232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.595210 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n5fx\" (UniqueName: \"kubernetes.io/projected/18bf49cc-fe19-48d9-8947-166ebcce843d-kube-api-access-5n5fx\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm\" (UID: \"18bf49cc-fe19-48d9-8947-166ebcce843d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" Oct 30 04:45:00.639143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.638213 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" Oct 30 04:45:00.675986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.674525 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw\" (UID: \"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" Oct 30 04:45:00.675986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.674694 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1fbc60-30e0-4846-b526-f52a9c8a4091-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc\" (UID: \"2c1fbc60-30e0-4846-b526-f52a9c8a4091\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" Oct 30 04:45:00.675986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.674742 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmcxs\" (UniqueName: \"kubernetes.io/projected/2c1fbc60-30e0-4846-b526-f52a9c8a4091-kube-api-access-vmcxs\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc\" (UID: \"2c1fbc60-30e0-4846-b526-f52a9c8a4091\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" Oct 30 04:45:00.675986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.674782 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5t9\" (UniqueName: \"kubernetes.io/projected/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-kube-api-access-fd5t9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw\" (UID: \"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" Oct 30 04:45:00.675986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.674810 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq98r\" (UniqueName: \"kubernetes.io/projected/d692e5e1-80df-47e2-9dc5-64adf498e349-kube-api-access-dq98r\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq\" (UID: \"d692e5e1-80df-47e2-9dc5-64adf498e349\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" Oct 30 04:45:00.675986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.674876 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d692e5e1-80df-47e2-9dc5-64adf498e349-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq\" (UID: \"d692e5e1-80df-47e2-9dc5-64adf498e349\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" Oct 30 04:45:00.675986 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.675725 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d692e5e1-80df-47e2-9dc5-64adf498e349-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq\" (UID: \"d692e5e1-80df-47e2-9dc5-64adf498e349\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" Oct 30 04:45:00.676622 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.676596 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1fbc60-30e0-4846-b526-f52a9c8a4091-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc\" (UID: \"2c1fbc60-30e0-4846-b526-f52a9c8a4091\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" Oct 30 04:45:00.676779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.676759 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw\" (UID: \"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" Oct 30 04:45:00.680817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.680751 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" Oct 30 04:45:00.703489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.702958 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq98r\" (UniqueName: \"kubernetes.io/projected/d692e5e1-80df-47e2-9dc5-64adf498e349-kube-api-access-dq98r\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq\" (UID: \"d692e5e1-80df-47e2-9dc5-64adf498e349\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" Oct 30 04:45:00.704295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.704247 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmcxs\" (UniqueName: \"kubernetes.io/projected/2c1fbc60-30e0-4846-b526-f52a9c8a4091-kube-api-access-vmcxs\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc\" (UID: \"2c1fbc60-30e0-4846-b526-f52a9c8a4091\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" Oct 30 04:45:00.707340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.707030 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5t9\" (UniqueName: \"kubernetes.io/projected/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-kube-api-access-fd5t9\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw\" (UID: \"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" Oct 30 04:45:00.731193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.730713 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" Oct 30 04:45:00.740229 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.739302 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" Oct 30 04:45:00.746093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.745911 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" Oct 30 04:45:00.904755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.904700 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5"] Oct 30 04:45:00.917293 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:45:00.917254 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3d37d2_eb6c_4dd1_a80a_bceb39880640.slice/crio-4b575ff3f979f67f6b610c70b0362b75f9acca869f9f6609ff58a0ff55e935d1 WatchSource:0}: Error finding container 4b575ff3f979f67f6b610c70b0362b75f9acca869f9f6609ff58a0ff55e935d1: Status 404 returned error can't find the container with id 4b575ff3f979f67f6b610c70b0362b75f9acca869f9f6609ff58a0ff55e935d1 Oct 30 04:45:00.983396 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:00.983367 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq"] Oct 30 04:45:00.997202 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:45:00.997169 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac757cc2_5436_41e3_b954_d3c0230757f9.slice/crio-a2a832013f7c0b0e88e9b482990dc63a784d6883da11dfbc4189408c17f2fc8a WatchSource:0}: Error finding container a2a832013f7c0b0e88e9b482990dc63a784d6883da11dfbc4189408c17f2fc8a: Status 404 returned error can't find the container with id a2a832013f7c0b0e88e9b482990dc63a784d6883da11dfbc4189408c17f2fc8a Oct 30 04:45:01.096738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.096696 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm"] Oct 30 04:45:01.097644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.097612 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:01.136053 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.136027 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc"] Oct 30 04:45:01.150305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.150284 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw"] Oct 30 04:45:01.165544 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.165518 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq"] Oct 30 04:45:01.322811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.322781 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:01.344811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.344779 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" event={"ID":"2c1fbc60-30e0-4846-b526-f52a9c8a4091","Type":"ContainerStarted","Data":"14cd7f66f5037a2945333d090c8b150f281a2cc5a5d42dfccb6804ec2fc6b795"} Oct 30 04:45:01.346067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.346035 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" event={"ID":"ac757cc2-5436-41e3-b954-d3c0230757f9","Type":"ContainerStarted","Data":"a2a832013f7c0b0e88e9b482990dc63a784d6883da11dfbc4189408c17f2fc8a"} Oct 30 04:45:01.347469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.347437 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" event={"ID":"18bf49cc-fe19-48d9-8947-166ebcce843d","Type":"ContainerStarted","Data":"c1466f6b1f824ffa4b318ede54ccf3d0a1dea817f6a1abbfbcb0e496ee6dbe8a"} Oct 30 04:45:01.348731 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.348708 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" event={"ID":"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0","Type":"ContainerStarted","Data":"cee29cc9930d569518bf7587bad43035dd763b5cc8a2c725b48b1e1fcdde343d"} Oct 30 04:45:01.349966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.349944 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" event={"ID":"d692e5e1-80df-47e2-9dc5-64adf498e349","Type":"ContainerStarted","Data":"c0bc4e06980ca42f91f695cbcbf1eec31c91519894bd63da25957277ef06d328"} Oct 30 04:45:01.351138 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.351119 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" event={"ID":"2c3d37d2-eb6c-4dd1-a80a-bceb39880640","Type":"ContainerStarted","Data":"4b575ff3f979f67f6b610c70b0362b75f9acca869f9f6609ff58a0ff55e935d1"} Oct 30 04:45:01.888575 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:01.888496 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:02.123170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.123144 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:02.362981 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.362948 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" event={"ID":"2c1fbc60-30e0-4846-b526-f52a9c8a4091","Type":"ContainerStarted","Data":"31b5bad426391c4ba09879d675e7b017b9bd7942b0cccbf5343559aec8a60f75"} Oct 30 04:45:02.363276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.363250 4793 scope.go:122] "RemoveContainer" containerID="31b5bad426391c4ba09879d675e7b017b9bd7942b0cccbf5343559aec8a60f75" Oct 30 04:45:02.364842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.364815 4793 generic.go:356] "Generic (PLEG): container finished" podID="ac757cc2-5436-41e3-b954-d3c0230757f9" containerID="b71ba0645c3e3291cacefb680e9dfa0201cd73f430cc8a5cfe3e1a1185fdec7c" exitCode=60 Oct 30 04:45:02.365053 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.365016 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" event={"ID":"ac757cc2-5436-41e3-b954-d3c0230757f9","Type":"ContainerDied","Data":"b71ba0645c3e3291cacefb680e9dfa0201cd73f430cc8a5cfe3e1a1185fdec7c"} Oct 30 04:45:02.365173 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.365151 4793 scope.go:122] "RemoveContainer" containerID="b71ba0645c3e3291cacefb680e9dfa0201cd73f430cc8a5cfe3e1a1185fdec7c" Oct 30 04:45:02.375938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.375429 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" event={"ID":"18bf49cc-fe19-48d9-8947-166ebcce843d","Type":"ContainerStarted","Data":"a283300578ad445f1b8b75945953ac31a79182a0917333938c618c31ddbc0fe3"} Oct 30 04:45:02.377173 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.376838 4793 scope.go:122] "RemoveContainer" containerID="a283300578ad445f1b8b75945953ac31a79182a0917333938c618c31ddbc0fe3" Oct 30 04:45:02.379952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.379917 4793 generic.go:356] "Generic (PLEG): container finished" podID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" containerID="ec9bb33f1c82388be22041755c45371ba7f207c314163e264ea021e7c4a48811" exitCode=60 Oct 30 04:45:02.380080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.380061 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" event={"ID":"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0","Type":"ContainerDied","Data":"ec9bb33f1c82388be22041755c45371ba7f207c314163e264ea021e7c4a48811"} Oct 30 04:45:02.380514 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.380483 4793 scope.go:122] "RemoveContainer" containerID="ec9bb33f1c82388be22041755c45371ba7f207c314163e264ea021e7c4a48811" Oct 30 04:45:02.394436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.394397 4793 generic.go:356] "Generic (PLEG): container finished" podID="d692e5e1-80df-47e2-9dc5-64adf498e349" containerID="667edbd74a7bb136581134a3fbdff699832c94262885a6fe093a05248fbb8165" exitCode=60 Oct 30 04:45:02.394678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.394653 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" event={"ID":"d692e5e1-80df-47e2-9dc5-64adf498e349","Type":"ContainerDied","Data":"667edbd74a7bb136581134a3fbdff699832c94262885a6fe093a05248fbb8165"} Oct 30 04:45:02.394909 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.394828 4793 scope.go:122] "RemoveContainer" containerID="667edbd74a7bb136581134a3fbdff699832c94262885a6fe093a05248fbb8165" Oct 30 04:45:02.397604 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.397578 4793 generic.go:356] "Generic (PLEG): container finished" podID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" containerID="f779301d4484ee7998f35c221d726bcab5f9e44395d15e598a865c06513e6890" exitCode=60 Oct 30 04:45:02.397713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.397657 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" event={"ID":"2c3d37d2-eb6c-4dd1-a80a-bceb39880640","Type":"ContainerDied","Data":"f779301d4484ee7998f35c221d726bcab5f9e44395d15e598a865c06513e6890"} Oct 30 04:45:02.397981 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.397962 4793 scope.go:122] "RemoveContainer" containerID="f779301d4484ee7998f35c221d726bcab5f9e44395d15e598a865c06513e6890" Oct 30 04:45:02.773381 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:02.773340 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:03.064457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.061635 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:03.409358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.409277 4793 generic.go:356] "Generic (PLEG): container finished" podID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" containerID="31b5bad426391c4ba09879d675e7b017b9bd7942b0cccbf5343559aec8a60f75" exitCode=60 Oct 30 04:45:03.409770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.409360 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" event={"ID":"2c1fbc60-30e0-4846-b526-f52a9c8a4091","Type":"ContainerDied","Data":"31b5bad426391c4ba09879d675e7b017b9bd7942b0cccbf5343559aec8a60f75"} Oct 30 04:45:03.409770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.409444 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" event={"ID":"2c1fbc60-30e0-4846-b526-f52a9c8a4091","Type":"ContainerStarted","Data":"77808ff07aed14ae9a61b762e342cb568f6743ae77bde9dd991380b482174dfa"} Oct 30 04:45:03.411599 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.411577 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" event={"ID":"ac757cc2-5436-41e3-b954-d3c0230757f9","Type":"ContainerStarted","Data":"1195e6e892d38e1d7225fd26e30d3f5c476bc927695dc079e0f948441e2357c0"} Oct 30 04:45:03.413499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.413478 4793 generic.go:356] "Generic (PLEG): container finished" podID="18bf49cc-fe19-48d9-8947-166ebcce843d" containerID="a283300578ad445f1b8b75945953ac31a79182a0917333938c618c31ddbc0fe3" exitCode=60 Oct 30 04:45:03.413609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.413561 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" event={"ID":"18bf49cc-fe19-48d9-8947-166ebcce843d","Type":"ContainerDied","Data":"a283300578ad445f1b8b75945953ac31a79182a0917333938c618c31ddbc0fe3"} Oct 30 04:45:03.413674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.413611 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" event={"ID":"18bf49cc-fe19-48d9-8947-166ebcce843d","Type":"ContainerStarted","Data":"17b2ae3922bfa9fb3639d8f576c5942fb74bdc0a8cd71e7d4b4dbdcda217f913"} Oct 30 04:45:03.415542 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.415522 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" event={"ID":"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0","Type":"ContainerStarted","Data":"ac8f977b33829006d000dcfd58ffe07873e2fcb708cabd56b8c5ae3455b282c6"} Oct 30 04:45:03.417275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.417252 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" event={"ID":"d692e5e1-80df-47e2-9dc5-64adf498e349","Type":"ContainerStarted","Data":"924f77106597c0bd9432b1bce93048f9af676a88771a5574a7758e8068fcdd97"} Oct 30 04:45:03.419114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.419092 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" event={"ID":"2c3d37d2-eb6c-4dd1-a80a-bceb39880640","Type":"ContainerStarted","Data":"267c7c2b2a6980f3eda2073bdc10238e2ba907344208a897994d66a18e4ef009"} Oct 30 04:45:03.454521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.454471 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" podStartSLOduration=3.454453911 podStartE2EDuration="3.454453911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:03.44938681 +0000 UTC m=+2826.735261584" watchObservedRunningTime="2026-10-30 04:45:03.454453911 +0000 UTC m=+2826.740328686" Oct 30 04:45:03.493634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.493585 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" podStartSLOduration=3.4935704960000002 podStartE2EDuration="3.493570496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:03.490113117 +0000 UTC m=+2826.775987916" watchObservedRunningTime="2026-10-30 04:45:03.493570496 +0000 UTC m=+2826.779445272" Oct 30 04:45:03.534169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.534124 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" podStartSLOduration=3.534110685 podStartE2EDuration="3.534110685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:03.530553998 +0000 UTC m=+2826.816428770" watchObservedRunningTime="2026-10-30 04:45:03.534110685 +0000 UTC m=+2826.819985460" Oct 30 04:45:03.577598 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.577549 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" podStartSLOduration=3.57753479 podStartE2EDuration="3.57753479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:03.575967829 +0000 UTC m=+2826.861842634" watchObservedRunningTime="2026-10-30 04:45:03.57753479 +0000 UTC m=+2826.863409607" Oct 30 04:45:03.622109 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.622052 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" podStartSLOduration=3.622038551 podStartE2EDuration="3.622038551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:03.619109325 +0000 UTC m=+2826.904984099" watchObservedRunningTime="2026-10-30 04:45:03.622038551 +0000 UTC m=+2826.907913384" Oct 30 04:45:03.663893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:03.663805 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" podStartSLOduration=3.663787597 podStartE2EDuration="3.663787597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:03.659317908 +0000 UTC m=+2826.945192688" watchObservedRunningTime="2026-10-30 04:45:03.663787597 +0000 UTC m=+2826.949662373" Oct 30 04:45:04.430802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.430769 4793 generic.go:356] "Generic (PLEG): container finished" podID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" containerID="77808ff07aed14ae9a61b762e342cb568f6743ae77bde9dd991380b482174dfa" exitCode=60 Oct 30 04:45:04.431271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.430854 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" event={"ID":"2c1fbc60-30e0-4846-b526-f52a9c8a4091","Type":"ContainerDied","Data":"77808ff07aed14ae9a61b762e342cb568f6743ae77bde9dd991380b482174dfa"} Oct 30 04:45:04.431271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.430930 4793 scope.go:122] "RemoveContainer" containerID="31b5bad426391c4ba09879d675e7b017b9bd7942b0cccbf5343559aec8a60f75" Oct 30 04:45:04.431271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.431086 4793 scope.go:122] "RemoveContainer" containerID="77808ff07aed14ae9a61b762e342cb568f6743ae77bde9dd991380b482174dfa" Oct 30 04:45:04.431495 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:04.431475 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc_odh-ai-gateway-infra(2c1fbc60-30e0-4846-b526-f52a9c8a4091)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" Oct 30 04:45:04.432891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.432864 4793 generic.go:356] "Generic (PLEG): container finished" podID="ac757cc2-5436-41e3-b954-d3c0230757f9" containerID="1195e6e892d38e1d7225fd26e30d3f5c476bc927695dc079e0f948441e2357c0" exitCode=60 Oct 30 04:45:04.433026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.432920 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" event={"ID":"ac757cc2-5436-41e3-b954-d3c0230757f9","Type":"ContainerDied","Data":"1195e6e892d38e1d7225fd26e30d3f5c476bc927695dc079e0f948441e2357c0"} Oct 30 04:45:04.433181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.433166 4793 scope.go:122] "RemoveContainer" containerID="1195e6e892d38e1d7225fd26e30d3f5c476bc927695dc079e0f948441e2357c0" Oct 30 04:45:04.433485 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:04.433459 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq_odh-ai-gateway-infra(ac757cc2-5436-41e3-b954-d3c0230757f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" Oct 30 04:45:04.435198 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.435177 4793 generic.go:356] "Generic (PLEG): container finished" podID="18bf49cc-fe19-48d9-8947-166ebcce843d" containerID="17b2ae3922bfa9fb3639d8f576c5942fb74bdc0a8cd71e7d4b4dbdcda217f913" exitCode=60 Oct 30 04:45:04.435289 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.435247 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" event={"ID":"18bf49cc-fe19-48d9-8947-166ebcce843d","Type":"ContainerDied","Data":"17b2ae3922bfa9fb3639d8f576c5942fb74bdc0a8cd71e7d4b4dbdcda217f913"} Oct 30 04:45:04.435556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.435541 4793 scope.go:122] "RemoveContainer" containerID="17b2ae3922bfa9fb3639d8f576c5942fb74bdc0a8cd71e7d4b4dbdcda217f913" Oct 30 04:45:04.435806 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:04.435788 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm_odh-ai-gateway-infra(18bf49cc-fe19-48d9-8947-166ebcce843d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" Oct 30 04:45:04.437376 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.437352 4793 generic.go:356] "Generic (PLEG): container finished" podID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" containerID="ac8f977b33829006d000dcfd58ffe07873e2fcb708cabd56b8c5ae3455b282c6" exitCode=60 Oct 30 04:45:04.437488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.437448 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" event={"ID":"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0","Type":"ContainerDied","Data":"ac8f977b33829006d000dcfd58ffe07873e2fcb708cabd56b8c5ae3455b282c6"} Oct 30 04:45:04.437662 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.437646 4793 scope.go:122] "RemoveContainer" containerID="ac8f977b33829006d000dcfd58ffe07873e2fcb708cabd56b8c5ae3455b282c6" Oct 30 04:45:04.437910 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:04.437889 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw_odh-ai-gateway-infra(cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" Oct 30 04:45:04.439613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.439591 4793 generic.go:356] "Generic (PLEG): container finished" podID="d692e5e1-80df-47e2-9dc5-64adf498e349" containerID="924f77106597c0bd9432b1bce93048f9af676a88771a5574a7758e8068fcdd97" exitCode=60 Oct 30 04:45:04.439710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.439667 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" event={"ID":"d692e5e1-80df-47e2-9dc5-64adf498e349","Type":"ContainerDied","Data":"924f77106597c0bd9432b1bce93048f9af676a88771a5574a7758e8068fcdd97"} Oct 30 04:45:04.439872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.439857 4793 scope.go:122] "RemoveContainer" containerID="924f77106597c0bd9432b1bce93048f9af676a88771a5574a7758e8068fcdd97" Oct 30 04:45:04.440118 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:04.440096 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq_odh-ai-gateway-infra(d692e5e1-80df-47e2-9dc5-64adf498e349)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" Oct 30 04:45:04.441375 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.441354 4793 generic.go:356] "Generic (PLEG): container finished" podID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" containerID="267c7c2b2a6980f3eda2073bdc10238e2ba907344208a897994d66a18e4ef009" exitCode=60 Oct 30 04:45:04.441509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.441493 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" event={"ID":"2c3d37d2-eb6c-4dd1-a80a-bceb39880640","Type":"ContainerDied","Data":"267c7c2b2a6980f3eda2073bdc10238e2ba907344208a897994d66a18e4ef009"} Oct 30 04:45:04.441739 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.441725 4793 scope.go:122] "RemoveContainer" containerID="267c7c2b2a6980f3eda2073bdc10238e2ba907344208a897994d66a18e4ef009" Oct 30 04:45:04.441937 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:04.441921 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5_odh-ai-gateway-infra(2c3d37d2-eb6c-4dd1-a80a-bceb39880640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" Oct 30 04:45:04.445174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.445157 4793 scope.go:122] "RemoveContainer" containerID="b71ba0645c3e3291cacefb680e9dfa0201cd73f430cc8a5cfe3e1a1185fdec7c" Oct 30 04:45:04.459836 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.459812 4793 scope.go:122] "RemoveContainer" containerID="a283300578ad445f1b8b75945953ac31a79182a0917333938c618c31ddbc0fe3" Oct 30 04:45:04.476131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.476113 4793 scope.go:122] "RemoveContainer" containerID="ec9bb33f1c82388be22041755c45371ba7f207c314163e264ea021e7c4a48811" Oct 30 04:45:04.494155 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.494135 4793 scope.go:122] "RemoveContainer" containerID="667edbd74a7bb136581134a3fbdff699832c94262885a6fe093a05248fbb8165" Oct 30 04:45:04.507155 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.507137 4793 scope.go:122] "RemoveContainer" containerID="f779301d4484ee7998f35c221d726bcab5f9e44395d15e598a865c06513e6890" Oct 30 04:45:04.860870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:04.860774 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:05.301450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:05.301389 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:05.450841 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:05.450811 4793 scope.go:122] "RemoveContainer" containerID="17b2ae3922bfa9fb3639d8f576c5942fb74bdc0a8cd71e7d4b4dbdcda217f913" Oct 30 04:45:05.451291 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:05.451171 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm_odh-ai-gateway-infra(18bf49cc-fe19-48d9-8947-166ebcce843d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" Oct 30 04:45:05.452646 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:05.452619 4793 scope.go:122] "RemoveContainer" containerID="ac8f977b33829006d000dcfd58ffe07873e2fcb708cabd56b8c5ae3455b282c6" Oct 30 04:45:05.452973 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:05.452943 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw_odh-ai-gateway-infra(cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" Oct 30 04:45:05.454602 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:05.454577 4793 scope.go:122] "RemoveContainer" containerID="924f77106597c0bd9432b1bce93048f9af676a88771a5574a7758e8068fcdd97" Oct 30 04:45:05.454901 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:05.454873 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq_odh-ai-gateway-infra(d692e5e1-80df-47e2-9dc5-64adf498e349)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" Oct 30 04:45:05.456358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:05.456328 4793 scope.go:122] "RemoveContainer" containerID="267c7c2b2a6980f3eda2073bdc10238e2ba907344208a897994d66a18e4ef009" Oct 30 04:45:05.456658 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:05.456630 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5_odh-ai-gateway-infra(2c3d37d2-eb6c-4dd1-a80a-bceb39880640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" Oct 30 04:45:05.458667 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:05.458627 4793 scope.go:122] "RemoveContainer" containerID="77808ff07aed14ae9a61b762e342cb568f6743ae77bde9dd991380b482174dfa" Oct 30 04:45:05.458887 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:05.458866 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc_odh-ai-gateway-infra(2c1fbc60-30e0-4846-b526-f52a9c8a4091)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" Oct 30 04:45:05.460343 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:05.460324 4793 scope.go:122] "RemoveContainer" containerID="1195e6e892d38e1d7225fd26e30d3f5c476bc927695dc079e0f948441e2357c0" Oct 30 04:45:05.460618 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:05.460588 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq_odh-ai-gateway-infra(ac757cc2-5436-41e3-b954-d3c0230757f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" Oct 30 04:45:05.543385 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:05.543341 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:06.816483 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:06.815284 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:07.111939 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:07.111814 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:07.380205 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:07.380104 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:09.011033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:09.010997 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:10.628325 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:10.628234 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:11.219895 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:11.219864 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:45:12.370558 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:12.370520 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:14.045598 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:14.045562 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:14.838571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:14.838539 4793 scope.go:122] "RemoveContainer" containerID="77808ff07aed14ae9a61b762e342cb568f6743ae77bde9dd991380b482174dfa" Oct 30 04:45:14.838781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:14.838716 4793 scope.go:122] "RemoveContainer" containerID="ac8f977b33829006d000dcfd58ffe07873e2fcb708cabd56b8c5ae3455b282c6" Oct 30 04:45:15.532932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:15.532895 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" event={"ID":"2c1fbc60-30e0-4846-b526-f52a9c8a4091","Type":"ContainerStarted","Data":"8e4e17194905893e52859c81bb24f24b4e1f82f496117d8e5cf17be01dda6252"} Oct 30 04:45:15.534923 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:15.534890 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" event={"ID":"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0","Type":"ContainerStarted","Data":"0efe078f64499c3329913a9551187e497cd9a0bd500fffc5bb4c2a57a3a046b9"} Oct 30 04:45:15.838757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:15.838725 4793 scope.go:122] "RemoveContainer" containerID="267c7c2b2a6980f3eda2073bdc10238e2ba907344208a897994d66a18e4ef009" Oct 30 04:45:15.881325 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:15.881298 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:16.542974 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.542943 4793 generic.go:356] "Generic (PLEG): container finished" podID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" containerID="8e4e17194905893e52859c81bb24f24b4e1f82f496117d8e5cf17be01dda6252" exitCode=60 Oct 30 04:45:16.543401 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.543020 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" event={"ID":"2c1fbc60-30e0-4846-b526-f52a9c8a4091","Type":"ContainerDied","Data":"8e4e17194905893e52859c81bb24f24b4e1f82f496117d8e5cf17be01dda6252"} Oct 30 04:45:16.543401 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.543089 4793 scope.go:122] "RemoveContainer" containerID="77808ff07aed14ae9a61b762e342cb568f6743ae77bde9dd991380b482174dfa" Oct 30 04:45:16.543565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.543471 4793 scope.go:122] "RemoveContainer" containerID="8e4e17194905893e52859c81bb24f24b4e1f82f496117d8e5cf17be01dda6252" Oct 30 04:45:16.543872 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:16.543849 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc_odh-ai-gateway-infra(2c1fbc60-30e0-4846-b526-f52a9c8a4091)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" Oct 30 04:45:16.545384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.545361 4793 generic.go:356] "Generic (PLEG): container finished" podID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" containerID="0efe078f64499c3329913a9551187e497cd9a0bd500fffc5bb4c2a57a3a046b9" exitCode=60 Oct 30 04:45:16.545523 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.545444 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" event={"ID":"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0","Type":"ContainerDied","Data":"0efe078f64499c3329913a9551187e497cd9a0bd500fffc5bb4c2a57a3a046b9"} Oct 30 04:45:16.545835 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.545816 4793 scope.go:122] "RemoveContainer" containerID="0efe078f64499c3329913a9551187e497cd9a0bd500fffc5bb4c2a57a3a046b9" Oct 30 04:45:16.546156 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:16.546125 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw_odh-ai-gateway-infra(cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" Oct 30 04:45:16.547475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.547453 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" event={"ID":"2c3d37d2-eb6c-4dd1-a80a-bceb39880640","Type":"ContainerStarted","Data":"646454aa17436ea51d64d3372790684772d13bd549a2a78da1169db7c2a55c95"} Oct 30 04:45:16.558744 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.558722 4793 scope.go:122] "RemoveContainer" containerID="ac8f977b33829006d000dcfd58ffe07873e2fcb708cabd56b8c5ae3455b282c6" Oct 30 04:45:16.647470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.647433 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc"] Oct 30 04:45:16.726293 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.726262 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw"] Oct 30 04:45:16.844044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.843971 4793 scope.go:122] "RemoveContainer" containerID="17b2ae3922bfa9fb3639d8f576c5942fb74bdc0a8cd71e7d4b4dbdcda217f913" Oct 30 04:45:16.844189 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:16.844084 4793 scope.go:122] "RemoveContainer" containerID="924f77106597c0bd9432b1bce93048f9af676a88771a5574a7758e8068fcdd97" Oct 30 04:45:17.551866 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.551823 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:17.560399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.560372 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" event={"ID":"18bf49cc-fe19-48d9-8947-166ebcce843d","Type":"ContainerStarted","Data":"1f186f8b5ffe4e084bca98ba8d1619a7c6c623d3071efa3481bf83e5a4d4e134"} Oct 30 04:45:17.564477 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.564452 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" event={"ID":"d692e5e1-80df-47e2-9dc5-64adf498e349","Type":"ContainerStarted","Data":"4a7326560a2af6765858f92c82b3e848563502d6696b2b43df6182ca7ee0a8f3"} Oct 30 04:45:17.566684 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.566662 4793 generic.go:356] "Generic (PLEG): container finished" podID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" containerID="646454aa17436ea51d64d3372790684772d13bd549a2a78da1169db7c2a55c95" exitCode=60 Oct 30 04:45:17.566773 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.566704 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" event={"ID":"2c3d37d2-eb6c-4dd1-a80a-bceb39880640","Type":"ContainerDied","Data":"646454aa17436ea51d64d3372790684772d13bd549a2a78da1169db7c2a55c95"} Oct 30 04:45:17.566773 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.566743 4793 scope.go:122] "RemoveContainer" containerID="267c7c2b2a6980f3eda2073bdc10238e2ba907344208a897994d66a18e4ef009" Oct 30 04:45:17.567076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.567060 4793 scope.go:122] "RemoveContainer" containerID="646454aa17436ea51d64d3372790684772d13bd549a2a78da1169db7c2a55c95" Oct 30 04:45:17.567400 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:17.567375 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5_odh-ai-gateway-infra(2c3d37d2-eb6c-4dd1-a80a-bceb39880640)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" Oct 30 04:45:17.640550 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.640506 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5"] Oct 30 04:45:17.764580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.763002 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" Oct 30 04:45:17.838899 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.838826 4793 scope.go:122] "RemoveContainer" containerID="1195e6e892d38e1d7225fd26e30d3f5c476bc927695dc079e0f948441e2357c0" Oct 30 04:45:17.876246 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.876219 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c1fbc60-30e0-4846-b526-f52a9c8a4091-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1fbc60-30e0-4846-b526-f52a9c8a4091-maas-api-service-ca\") pod \"2c1fbc60-30e0-4846-b526-f52a9c8a4091\" (UID: \"2c1fbc60-30e0-4846-b526-f52a9c8a4091\") " Oct 30 04:45:17.876358 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.876298 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1fbc60-30e0-4846-b526-f52a9c8a4091-kube-api-access-vmcxs\" (UniqueName: \"kubernetes.io/projected/2c1fbc60-30e0-4846-b526-f52a9c8a4091-kube-api-access-vmcxs\") pod \"2c1fbc60-30e0-4846-b526-f52a9c8a4091\" (UID: \"2c1fbc60-30e0-4846-b526-f52a9c8a4091\") " Oct 30 04:45:17.876706 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.876683 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1fbc60-30e0-4846-b526-f52a9c8a4091-maas-api-service-ca" pod "2c1fbc60-30e0-4846-b526-f52a9c8a4091" (UID: "2c1fbc60-30e0-4846-b526-f52a9c8a4091"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:45:17.877057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.877031 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c1fbc60-30e0-4846-b526-f52a9c8a4091-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:45:17.879006 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.878905 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1fbc60-30e0-4846-b526-f52a9c8a4091-kube-api-access-vmcxs" pod "2c1fbc60-30e0-4846-b526-f52a9c8a4091" (UID: "2c1fbc60-30e0-4846-b526-f52a9c8a4091"). InnerVolumeSpecName "kube-api-access-vmcxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:45:17.979168 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:17.979136 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmcxs\" (UniqueName: \"kubernetes.io/projected/2c1fbc60-30e0-4846-b526-f52a9c8a4091-kube-api-access-vmcxs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:45:18.045975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.045954 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" Oct 30 04:45:18.181038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.181007 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-kube-api-access-fd5t9\" (UniqueName: \"kubernetes.io/projected/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-kube-api-access-fd5t9\") pod \"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0\" (UID: \"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0\") " Oct 30 04:45:18.181204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.181064 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-maas-api-service-ca\") pod \"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0\" (UID: \"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0\") " Oct 30 04:45:18.181394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.181369 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-maas-api-service-ca" pod "cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" (UID: "cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:45:18.181542 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.181527 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:45:18.183149 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.183130 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-kube-api-access-fd5t9" pod "cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" (UID: "cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0"). InnerVolumeSpecName "kube-api-access-fd5t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:45:18.282730 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.282708 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd5t9\" (UniqueName: \"kubernetes.io/projected/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0-kube-api-access-fd5t9\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:45:18.575476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.575413 4793 generic.go:356] "Generic (PLEG): container finished" podID="18bf49cc-fe19-48d9-8947-166ebcce843d" containerID="1f186f8b5ffe4e084bca98ba8d1619a7c6c623d3071efa3481bf83e5a4d4e134" exitCode=60 Oct 30 04:45:18.575936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.575486 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" event={"ID":"18bf49cc-fe19-48d9-8947-166ebcce843d","Type":"ContainerDied","Data":"1f186f8b5ffe4e084bca98ba8d1619a7c6c623d3071efa3481bf83e5a4d4e134"} Oct 30 04:45:18.575936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.575553 4793 scope.go:122] "RemoveContainer" containerID="17b2ae3922bfa9fb3639d8f576c5942fb74bdc0a8cd71e7d4b4dbdcda217f913" Oct 30 04:45:18.575936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.575917 4793 scope.go:122] "RemoveContainer" containerID="1f186f8b5ffe4e084bca98ba8d1619a7c6c623d3071efa3481bf83e5a4d4e134" Oct 30 04:45:18.576290 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:18.576266 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm_odh-ai-gateway-infra(18bf49cc-fe19-48d9-8947-166ebcce843d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" Oct 30 04:45:18.577103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.577082 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" Oct 30 04:45:18.577215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.577084 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw" event={"ID":"cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0","Type":"ContainerDied","Data":"cee29cc9930d569518bf7587bad43035dd763b5cc8a2c725b48b1e1fcdde343d"} Oct 30 04:45:18.579222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.579198 4793 generic.go:356] "Generic (PLEG): container finished" podID="d692e5e1-80df-47e2-9dc5-64adf498e349" containerID="4a7326560a2af6765858f92c82b3e848563502d6696b2b43df6182ca7ee0a8f3" exitCode=60 Oct 30 04:45:18.579320 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.579244 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" event={"ID":"d692e5e1-80df-47e2-9dc5-64adf498e349","Type":"ContainerDied","Data":"4a7326560a2af6765858f92c82b3e848563502d6696b2b43df6182ca7ee0a8f3"} Oct 30 04:45:18.579635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.579612 4793 scope.go:122] "RemoveContainer" containerID="4a7326560a2af6765858f92c82b3e848563502d6696b2b43df6182ca7ee0a8f3" Oct 30 04:45:18.579885 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:18.579857 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq_odh-ai-gateway-infra(d692e5e1-80df-47e2-9dc5-64adf498e349)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" Oct 30 04:45:18.582794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.582776 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" Oct 30 04:45:18.582882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.582814 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc" event={"ID":"2c1fbc60-30e0-4846-b526-f52a9c8a4091","Type":"ContainerDied","Data":"14cd7f66f5037a2945333d090c8b150f281a2cc5a5d42dfccb6804ec2fc6b795"} Oct 30 04:45:18.585109 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.585080 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" event={"ID":"ac757cc2-5436-41e3-b954-d3c0230757f9","Type":"ContainerStarted","Data":"f8e4c5806004fba57a202fe5a679abbf2d4f52e6a397b3fff7329d6c028fd3c0"} Oct 30 04:45:18.598429 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.598396 4793 scope.go:122] "RemoveContainer" containerID="0efe078f64499c3329913a9551187e497cd9a0bd500fffc5bb4c2a57a3a046b9" Oct 30 04:45:18.609130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.609110 4793 scope.go:122] "RemoveContainer" containerID="924f77106597c0bd9432b1bce93048f9af676a88771a5574a7758e8068fcdd97" Oct 30 04:45:18.629665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.629647 4793 scope.go:122] "RemoveContainer" containerID="8e4e17194905893e52859c81bb24f24b4e1f82f496117d8e5cf17be01dda6252" Oct 30 04:45:18.710235 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.710207 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw"] Oct 30 04:45:18.715089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.715058 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-66217b-29888925-lbvdw"] Oct 30 04:45:18.743677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.743647 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" Oct 30 04:45:18.746431 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.746393 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm"] Oct 30 04:45:18.792663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.792637 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq"] Oct 30 04:45:18.795917 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.795893 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc"] Oct 30 04:45:18.806161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.806137 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-fcd6a6-29888925-s8vnc"] Oct 30 04:45:18.847317 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.847244 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" path="/var/lib/kubelet/pods/2c1fbc60-30e0-4846-b526-f52a9c8a4091/volumes" Oct 30 04:45:18.847701 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.847685 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" path="/var/lib/kubelet/pods/cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0/volumes" Oct 30 04:45:18.889934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.889908 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-maas-api-service-ca\") pod \"2c3d37d2-eb6c-4dd1-a80a-bceb39880640\" (UID: \"2c3d37d2-eb6c-4dd1-a80a-bceb39880640\") " Oct 30 04:45:18.890039 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.890026 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-kube-api-access-blpw6\" (UniqueName: \"kubernetes.io/projected/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-kube-api-access-blpw6\") pod \"2c3d37d2-eb6c-4dd1-a80a-bceb39880640\" (UID: \"2c3d37d2-eb6c-4dd1-a80a-bceb39880640\") " Oct 30 04:45:18.890295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.890270 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-maas-api-service-ca" pod "2c3d37d2-eb6c-4dd1-a80a-bceb39880640" (UID: "2c3d37d2-eb6c-4dd1-a80a-bceb39880640"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:45:18.890629 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.890611 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:45:18.892031 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.892010 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-kube-api-access-blpw6" pod "2c3d37d2-eb6c-4dd1-a80a-bceb39880640" (UID: "2c3d37d2-eb6c-4dd1-a80a-bceb39880640"). InnerVolumeSpecName "kube-api-access-blpw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:45:18.992146 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:18.992114 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blpw6\" (UniqueName: \"kubernetes.io/projected/2c3d37d2-eb6c-4dd1-a80a-bceb39880640-kube-api-access-blpw6\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:45:19.342097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.342058 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:19.595883 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.595797 4793 generic.go:356] "Generic (PLEG): container finished" podID="ac757cc2-5436-41e3-b954-d3c0230757f9" containerID="f8e4c5806004fba57a202fe5a679abbf2d4f52e6a397b3fff7329d6c028fd3c0" exitCode=60 Oct 30 04:45:19.596245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.595982 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" event={"ID":"ac757cc2-5436-41e3-b954-d3c0230757f9","Type":"ContainerDied","Data":"f8e4c5806004fba57a202fe5a679abbf2d4f52e6a397b3fff7329d6c028fd3c0"} Oct 30 04:45:19.596245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.596074 4793 scope.go:122] "RemoveContainer" containerID="1195e6e892d38e1d7225fd26e30d3f5c476bc927695dc079e0f948441e2357c0" Oct 30 04:45:19.596608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.596588 4793 scope.go:122] "RemoveContainer" containerID="f8e4c5806004fba57a202fe5a679abbf2d4f52e6a397b3fff7329d6c028fd3c0" Oct 30 04:45:19.596991 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:19.596965 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq_odh-ai-gateway-infra(ac757cc2-5436-41e3-b954-d3c0230757f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" Oct 30 04:45:19.604940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.604918 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" Oct 30 04:45:19.605041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.604935 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5" event={"ID":"2c3d37d2-eb6c-4dd1-a80a-bceb39880640","Type":"ContainerDied","Data":"4b575ff3f979f67f6b610c70b0362b75f9acca869f9f6609ff58a0ff55e935d1"} Oct 30 04:45:19.614212 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.614196 4793 scope.go:122] "RemoveContainer" containerID="646454aa17436ea51d64d3372790684772d13bd549a2a78da1169db7c2a55c95" Oct 30 04:45:19.798894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.798831 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5"] Oct 30 04:45:19.803802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.803771 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6796b1-29888925-49gr5"] Oct 30 04:45:19.809465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.809437 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" Oct 30 04:45:19.838038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.838015 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" Oct 30 04:45:19.865580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.865507 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq"] Oct 30 04:45:19.905480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.905449 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18bf49cc-fe19-48d9-8947-166ebcce843d-kube-api-access-5n5fx\" (UniqueName: \"kubernetes.io/projected/18bf49cc-fe19-48d9-8947-166ebcce843d-kube-api-access-5n5fx\") pod \"18bf49cc-fe19-48d9-8947-166ebcce843d\" (UID: \"18bf49cc-fe19-48d9-8947-166ebcce843d\") " Oct 30 04:45:19.905652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.905561 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18bf49cc-fe19-48d9-8947-166ebcce843d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18bf49cc-fe19-48d9-8947-166ebcce843d-maas-api-service-ca\") pod \"18bf49cc-fe19-48d9-8947-166ebcce843d\" (UID: \"18bf49cc-fe19-48d9-8947-166ebcce843d\") " Oct 30 04:45:19.906046 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.906017 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18bf49cc-fe19-48d9-8947-166ebcce843d-maas-api-service-ca" pod "18bf49cc-fe19-48d9-8947-166ebcce843d" (UID: "18bf49cc-fe19-48d9-8947-166ebcce843d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:45:19.906342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.906307 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18bf49cc-fe19-48d9-8947-166ebcce843d-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:45:19.907984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:19.907955 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18bf49cc-fe19-48d9-8947-166ebcce843d-kube-api-access-5n5fx" pod "18bf49cc-fe19-48d9-8947-166ebcce843d" (UID: "18bf49cc-fe19-48d9-8947-166ebcce843d"). InnerVolumeSpecName "kube-api-access-5n5fx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:45:20.007712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.007676 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d692e5e1-80df-47e2-9dc5-64adf498e349-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d692e5e1-80df-47e2-9dc5-64adf498e349-maas-api-service-ca\") pod \"d692e5e1-80df-47e2-9dc5-64adf498e349\" (UID: \"d692e5e1-80df-47e2-9dc5-64adf498e349\") " Oct 30 04:45:20.007868 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.007741 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d692e5e1-80df-47e2-9dc5-64adf498e349-kube-api-access-dq98r\" (UniqueName: \"kubernetes.io/projected/d692e5e1-80df-47e2-9dc5-64adf498e349-kube-api-access-dq98r\") pod \"d692e5e1-80df-47e2-9dc5-64adf498e349\" (UID: \"d692e5e1-80df-47e2-9dc5-64adf498e349\") " Oct 30 04:45:20.008302 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.008281 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n5fx\" (UniqueName: \"kubernetes.io/projected/18bf49cc-fe19-48d9-8947-166ebcce843d-kube-api-access-5n5fx\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:45:20.008769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.008743 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d692e5e1-80df-47e2-9dc5-64adf498e349-maas-api-service-ca" pod "d692e5e1-80df-47e2-9dc5-64adf498e349" (UID: "d692e5e1-80df-47e2-9dc5-64adf498e349"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:45:20.010531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.010507 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d692e5e1-80df-47e2-9dc5-64adf498e349-kube-api-access-dq98r" pod "d692e5e1-80df-47e2-9dc5-64adf498e349" (UID: "d692e5e1-80df-47e2-9dc5-64adf498e349"). InnerVolumeSpecName "kube-api-access-dq98r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:45:20.109806 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.109778 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d692e5e1-80df-47e2-9dc5-64adf498e349-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:45:20.109806 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.109805 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq98r\" (UniqueName: \"kubernetes.io/projected/d692e5e1-80df-47e2-9dc5-64adf498e349-kube-api-access-dq98r\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:45:20.618158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.618129 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" Oct 30 04:45:20.618158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.618148 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm" event={"ID":"18bf49cc-fe19-48d9-8947-166ebcce843d","Type":"ContainerDied","Data":"c1466f6b1f824ffa4b318ede54ccf3d0a1dea817f6a1abbfbcb0e496ee6dbe8a"} Oct 30 04:45:20.618655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.618209 4793 scope.go:122] "RemoveContainer" containerID="1f186f8b5ffe4e084bca98ba8d1619a7c6c623d3071efa3481bf83e5a4d4e134" Oct 30 04:45:20.619691 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.619666 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" Oct 30 04:45:20.619820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.619661 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq" event={"ID":"d692e5e1-80df-47e2-9dc5-64adf498e349","Type":"ContainerDied","Data":"c0bc4e06980ca42f91f695cbcbf1eec31c91519894bd63da25957277ef06d328"} Oct 30 04:45:20.628538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.628510 4793 scope.go:122] "RemoveContainer" containerID="4a7326560a2af6765858f92c82b3e848563502d6696b2b43df6182ca7ee0a8f3" Oct 30 04:45:20.701967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.701939 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm"] Oct 30 04:45:20.706213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.706189 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888925-6lwgm"] Oct 30 04:45:20.745851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.745823 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq"] Oct 30 04:45:20.752013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.751991 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" Oct 30 04:45:20.752462 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.752437 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888925-qjgsq"] Oct 30 04:45:20.847215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.847182 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" path="/var/lib/kubelet/pods/18bf49cc-fe19-48d9-8947-166ebcce843d/volumes" Oct 30 04:45:20.847666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.847652 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" path="/var/lib/kubelet/pods/2c3d37d2-eb6c-4dd1-a80a-bceb39880640/volumes" Oct 30 04:45:20.848049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.848037 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" path="/var/lib/kubelet/pods/d692e5e1-80df-47e2-9dc5-64adf498e349/volumes" Oct 30 04:45:20.920519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.920445 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac757cc2-5436-41e3-b954-d3c0230757f9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac757cc2-5436-41e3-b954-d3c0230757f9-maas-api-service-ca\") pod \"ac757cc2-5436-41e3-b954-d3c0230757f9\" (UID: \"ac757cc2-5436-41e3-b954-d3c0230757f9\") " Oct 30 04:45:20.920673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.920567 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac757cc2-5436-41e3-b954-d3c0230757f9-kube-api-access-vb84g\" (UniqueName: \"kubernetes.io/projected/ac757cc2-5436-41e3-b954-d3c0230757f9-kube-api-access-vb84g\") pod \"ac757cc2-5436-41e3-b954-d3c0230757f9\" (UID: \"ac757cc2-5436-41e3-b954-d3c0230757f9\") " Oct 30 04:45:20.920801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.920778 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac757cc2-5436-41e3-b954-d3c0230757f9-maas-api-service-ca" pod "ac757cc2-5436-41e3-b954-d3c0230757f9" (UID: "ac757cc2-5436-41e3-b954-d3c0230757f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:45:20.921008 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.920992 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac757cc2-5436-41e3-b954-d3c0230757f9-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:45:20.922681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:20.922655 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac757cc2-5436-41e3-b954-d3c0230757f9-kube-api-access-vb84g" pod "ac757cc2-5436-41e3-b954-d3c0230757f9" (UID: "ac757cc2-5436-41e3-b954-d3c0230757f9"). InnerVolumeSpecName "kube-api-access-vb84g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:45:21.022921 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:21.022884 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb84g\" (UniqueName: \"kubernetes.io/projected/ac757cc2-5436-41e3-b954-d3c0230757f9-kube-api-access-vb84g\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:45:21.130191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:21.130154 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:21.628940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:21.628909 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" Oct 30 04:45:21.628940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:21.628925 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq" event={"ID":"ac757cc2-5436-41e3-b954-d3c0230757f9","Type":"ContainerDied","Data":"a2a832013f7c0b0e88e9b482990dc63a784d6883da11dfbc4189408c17f2fc8a"} Oct 30 04:45:21.629496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:21.628981 4793 scope.go:122] "RemoveContainer" containerID="f8e4c5806004fba57a202fe5a679abbf2d4f52e6a397b3fff7329d6c028fd3c0" Oct 30 04:45:21.688812 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:21.688786 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq"] Oct 30 04:45:21.701371 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:21.701347 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-ed8266-29888925-jt2gq"] Oct 30 04:45:22.772511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:22.771873 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:22.847867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:22.847836 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" path="/var/lib/kubelet/pods/ac757cc2-5436-41e3-b954-d3c0230757f9/volumes" Oct 30 04:45:23.011106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:23.011067 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:24.513074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:24.513027 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:26.076998 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.076963 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782"] Oct 30 04:45:26.077593 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077576 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077593 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077605 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077610 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077623 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077628 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077640 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077646 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077652 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077657 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077668 4793 cpu_manager.go:414] "containerMap: removing container" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077673 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077678 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077682 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077690 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" containerName="cleanup" Oct 30 04:45:26.077686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077695 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077702 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077706 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077717 4793 cpu_manager.go:414] "containerMap: removing container" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077722 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077727 4793 cpu_manager.go:414] "containerMap: removing container" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077732 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077745 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077749 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077761 4793 cpu_manager.go:414] "containerMap: removing container" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077765 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077778 4793 cpu_manager.go:414] "containerMap: removing container" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077782 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077865 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077873 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077877 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077886 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077892 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077898 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3d37d2-eb6c-4dd1-a80a-bceb39880640" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077904 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077908 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077912 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077918 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077923 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077929 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077935 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077941 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" containerName="cleanup" Oct 30 04:45:26.078782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.077948 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="18bf49cc-fe19-48d9-8947-166ebcce843d" containerName="cleanup" Oct 30 04:45:26.082901 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.082878 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.103203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.103176 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782"] Oct 30 04:45:26.109835 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.109808 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-483bea-serving-cert\"" Oct 30 04:45:26.189428 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.189385 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-metrics-certs\") pod \"maas-api-e2e-extmod-b-483bea-6f457495fd-64782\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.189610 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.189518 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-483bea-6f457495fd-64782\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.189683 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.189621 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wc4k\" (UniqueName: \"kubernetes.io/projected/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-kube-api-access-6wc4k\") pod \"maas-api-e2e-extmod-b-483bea-6f457495fd-64782\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.189796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.189780 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-483bea-6f457495fd-64782\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.291997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.290258 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:26.291997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.291102 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wc4k\" (UniqueName: \"kubernetes.io/projected/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-kube-api-access-6wc4k\") pod \"maas-api-e2e-extmod-b-483bea-6f457495fd-64782\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.291997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.291231 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-483bea-6f457495fd-64782\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.291997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.291356 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-metrics-certs\") pod \"maas-api-e2e-extmod-b-483bea-6f457495fd-64782\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.291997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.291401 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-483bea-6f457495fd-64782\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.302491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.295481 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-483bea-6f457495fd-64782\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.302491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.296518 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-483bea-6f457495fd-64782\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.302491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.301919 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-metrics-certs\") pod \"maas-api-e2e-extmod-b-483bea-6f457495fd-64782\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.319079 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.319053 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wc4k\" (UniqueName: \"kubernetes.io/projected/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-kube-api-access-6wc4k\") pod \"maas-api-e2e-extmod-b-483bea-6f457495fd-64782\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.397807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.397735 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:26.511077 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.511033 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:26.702517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.702482 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782"] Oct 30 04:45:26.707103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.707080 4793 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:45:26.713940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:26.713913 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" event={"ID":"3049a33a-cac2-4174-82d4-7e0a7b96fdb0","Type":"ContainerStarted","Data":"5fa3ab84918727e4e907f1e5dbcdd24730b5a518a4331ae4e86323639e1d615b"} Oct 30 04:45:27.725682 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:27.725627 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" event={"ID":"3049a33a-cac2-4174-82d4-7e0a7b96fdb0","Type":"ContainerStarted","Data":"db300770b387802d795c0985eaee803570a8b259fb6c7e89ca0ad8e05569bc44"} Oct 30 04:45:27.726181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:27.725709 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:27.769702 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:27.769642 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" podStartSLOduration=1.375970087 podStartE2EDuration="1.769623706s" totalImagesPullingTime="393.653619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:27.768491579 +0000 UTC m=+2851.054366390" watchObservedRunningTime="2026-10-30 04:45:27.769623706 +0000 UTC m=+2851.055498486" Oct 30 04:45:28.129568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:28.129490 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:29.876212 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:29.876174 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:31.289502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.289471 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv"] Oct 30 04:45:31.290196 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.290178 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" containerName="cleanup" Oct 30 04:45:31.290270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.290198 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" containerName="cleanup" Oct 30 04:45:31.290270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.290216 4793 cpu_manager.go:414] "containerMap: removing container" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" containerName="cleanup" Oct 30 04:45:31.290270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.290221 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbecdbd8-5e82-4c7e-b3e3-b5070dc848b0" containerName="cleanup" Oct 30 04:45:31.290270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.290240 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" containerName="cleanup" Oct 30 04:45:31.290270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.290246 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" containerName="cleanup" Oct 30 04:45:31.290467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.290273 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" containerName="cleanup" Oct 30 04:45:31.290467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.290279 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" containerName="cleanup" Oct 30 04:45:31.290467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.290391 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1fbc60-30e0-4846-b526-f52a9c8a4091" containerName="cleanup" Oct 30 04:45:31.290467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.290402 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d692e5e1-80df-47e2-9dc5-64adf498e349" containerName="cleanup" Oct 30 04:45:31.290467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.290427 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac757cc2-5436-41e3-b954-d3c0230757f9" containerName="cleanup" Oct 30 04:45:31.294162 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.294141 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.313633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.313607 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv"] Oct 30 04:45:31.321822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.321796 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-05bee8-serving-cert\"" Oct 30 04:45:31.394508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.394478 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-metrics-certs\") pod \"maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.394714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.394617 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qlhv\" (UniqueName: \"kubernetes.io/projected/a19cf987-465a-4521-b6d9-50a9f27a9b7a-kube-api-access-2qlhv\") pod \"maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.394796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.394748 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19cf987-465a-4521-b6d9-50a9f27a9b7a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.394796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.394771 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.483316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.483276 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:31.504112 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.496533 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-metrics-certs\") pod \"maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.504699 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.504645 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qlhv\" (UniqueName: \"kubernetes.io/projected/a19cf987-465a-4521-b6d9-50a9f27a9b7a-kube-api-access-2qlhv\") pod \"maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.506209 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.506118 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19cf987-465a-4521-b6d9-50a9f27a9b7a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.506575 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.506554 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.509109 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.509082 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19cf987-465a-4521-b6d9-50a9f27a9b7a-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.516033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.516008 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-maas-api-tls\") pod \"maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.516173 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.516029 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-metrics-certs\") pod \"maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.533799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.533765 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qlhv\" (UniqueName: \"kubernetes.io/projected/a19cf987-465a-4521-b6d9-50a9f27a9b7a-kube-api-access-2qlhv\") pod \"maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.613817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.613741 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:31.703877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.703846 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:31.917971 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:45:31.917943 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19cf987_465a_4521_b6d9_50a9f27a9b7a.slice/crio-7836e94465e9bb6200454b35801c7791ba6bf147bee33d31b2b82b7fe13183bb WatchSource:0}: Error finding container 7836e94465e9bb6200454b35801c7791ba6bf147bee33d31b2b82b7fe13183bb: Status 404 returned error can't find the container with id 7836e94465e9bb6200454b35801c7791ba6bf147bee33d31b2b82b7fe13183bb Oct 30 04:45:31.918186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:31.918163 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv"] Oct 30 04:45:32.762332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:32.762295 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" event={"ID":"a19cf987-465a-4521-b6d9-50a9f27a9b7a","Type":"ContainerStarted","Data":"92e393dc8cbe2fb08e6a1f78ae9068b111719a2cb2d7d12b9614779c618afa9c"} Oct 30 04:45:32.762756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:32.762353 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" event={"ID":"a19cf987-465a-4521-b6d9-50a9f27a9b7a","Type":"ContainerStarted","Data":"7836e94465e9bb6200454b35801c7791ba6bf147bee33d31b2b82b7fe13183bb"} Oct 30 04:45:32.762756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:32.762442 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:32.808628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:32.808579 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" podStartSLOduration=1.453410475 podStartE2EDuration="1.8085624s" totalImagesPullingTime="355.151925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:45:32.807155315 +0000 UTC m=+2856.093030089" watchObservedRunningTime="2026-10-30 04:45:32.8085624 +0000 UTC m=+2856.094437187" Oct 30 04:45:33.235078 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:33.235045 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:33.738375 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:33.738342 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 04:45:35.020164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:35.020127 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:36.675650 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:36.675555 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:36.885714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:36.885681 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:38.326512 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:38.326477 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:38.774437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:38.774399 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 04:45:40.006780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:40.006745 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:41.219744 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:45:41.219704 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:45:41.712167 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:41.712136 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:43.448967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:43.448927 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:45.229368 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:45.229334 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:46.907054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:46.907019 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:48.510011 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:48.509971 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:50.138443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:50.138224 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:51.811658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:51.811621 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:53.464969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:53.464934 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:54.216880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:54.216852 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 04:45:54.217062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:54.216942 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 04:45:54.366804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:54.366712 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:54.605375 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:54.605340 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:55.034790 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.034754 4793 generic.go:356] "Generic (PLEG): container finished" podID="07de9748-a77d-4181-8202-6edb393a4882" containerID="ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba" exitCode=0 Oct 30 04:45:55.034994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.034827 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" event={"ID":"07de9748-a77d-4181-8202-6edb393a4882","Type":"ContainerDied","Data":"ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba"} Oct 30 04:45:55.035249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.035230 4793 scope.go:122] "RemoveContainer" containerID="ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba" Oct 30 04:45:55.036782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.036761 4793 generic.go:356] "Generic (PLEG): container finished" podID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerID="026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56" exitCode=0 Oct 30 04:45:55.036895 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.036841 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" event={"ID":"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e","Type":"ContainerDied","Data":"026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56"} Oct 30 04:45:55.037082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.037055 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:45:55.037162 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.037141 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:45:55.037230 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.037202 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:45:55.037287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.037246 4793 scope.go:122] "RemoveContainer" containerID="026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56" Oct 30 04:45:55.038953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.038921 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:45:55.039028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.039006 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:45:55.039070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.039057 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:45:55.135977 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.135832 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:55.229773 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.226783 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:45:55.734524 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:55.734488 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:56.046669 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:56.046582 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" event={"ID":"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e","Type":"ContainerStarted","Data":"5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b"} Oct 30 04:45:56.048757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:56.048727 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" event={"ID":"07de9748-a77d-4181-8202-6edb393a4882","Type":"ContainerStarted","Data":"815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682"} Oct 30 04:45:56.433441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:56.433402 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:45:56.434601 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:56.434574 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 04:45:56.434677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:56.434629 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 04:45:56.442771 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:56.442749 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:45:56.442929 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:56.442912 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 04:45:56.442984 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:56.442949 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 04:45:56.840831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:56.836852 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:57.434497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:57.434455 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 04:45:57.434690 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:57.434516 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 04:45:57.443408 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:57.443383 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 04:45:57.443547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:57.443439 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 04:45:58.433125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:58.433088 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 04:45:58.433556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:58.433142 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 04:45:58.443155 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:58.443127 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 04:45:58.443297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:58.443172 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 04:45:58.554442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:58.553638 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:45:59.433273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:59.433237 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 04:45:59.433695 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:59.433293 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 04:45:59.442689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:59.442662 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 04:45:59.442816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:45:59.442701 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 04:46:00.232663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:00.232275 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:00.433140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:00.433094 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 04:46:00.433348 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:00.433149 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 04:46:00.442620 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:00.442590 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" start-of-body= Oct 30 04:46:00.442758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:00.442630 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.193:15021/healthz/ready\": dial tcp 10.133.0.193:15021: connect: connection refused" Oct 30 04:46:01.673147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:01.672953 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:46:01.674054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:01.673945 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:46:01.675023 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:01.675000 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 04:46:01.706363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:01.706339 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:46:01.706752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:01.706717 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:46:01.708330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:01.708304 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 04:46:02.025567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:02.025524 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:02.248063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:02.248031 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:03.091602 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:03.091566 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:03.526607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:03.526573 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:03.754210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:03.754177 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:04.049840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:04.038066 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:04.283476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:04.283438 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:04.503581 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:04.503541 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:04.738082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:04.738052 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:04.953345 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:04.953307 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:05.260508 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:05.258187 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:05.555083 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:05.554988 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:05.891893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:05.891810 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:06.118673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:06.118636 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:06.770311 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:06.770274 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:07.345342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:07.345313 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:07.645070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:07.644992 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:07.865518 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:07.865482 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:08.694571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:08.694538 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:09.303921 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:09.303883 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:09.670489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:09.669967 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:09.901873 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:09.901835 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:11.219022 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:46:11.218983 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:46:11.460888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:11.460849 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:12.712271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:12.712236 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:12.928613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:12.928578 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:13.349528 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:13.349494 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:14.431985 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.431948 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:14.510936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.510904 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs"] Oct 30 04:46:14.514568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.514548 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" Oct 30 04:46:14.532994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.532970 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs"] Oct 30 04:46:14.541309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.541274 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-b195d0-dockercfg-jxfcn\"" Oct 30 04:46:14.560987 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.560964 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-b195d0\"" Oct 30 04:46:14.657448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.652305 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:14.685498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.684508 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhcrp\" (UniqueName: \"kubernetes.io/projected/d4873c9c-f109-4fb1-9653-77b12a7bba1e-kube-api-access-dhcrp\") pod \"payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs\" (UID: \"d4873c9c-f109-4fb1-9653-77b12a7bba1e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" Oct 30 04:46:14.685498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.684665 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4873c9c-f109-4fb1-9653-77b12a7bba1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs\" (UID: \"d4873c9c-f109-4fb1-9653-77b12a7bba1e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" Oct 30 04:46:14.785876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.785841 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhcrp\" (UniqueName: \"kubernetes.io/projected/d4873c9c-f109-4fb1-9653-77b12a7bba1e-kube-api-access-dhcrp\") pod \"payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs\" (UID: \"d4873c9c-f109-4fb1-9653-77b12a7bba1e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" Oct 30 04:46:14.786073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.785971 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4873c9c-f109-4fb1-9653-77b12a7bba1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs\" (UID: \"d4873c9c-f109-4fb1-9653-77b12a7bba1e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" Oct 30 04:46:14.786747 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.786728 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4873c9c-f109-4fb1-9653-77b12a7bba1e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs\" (UID: \"d4873c9c-f109-4fb1-9653-77b12a7bba1e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" Oct 30 04:46:14.809700 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.809674 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhcrp\" (UniqueName: \"kubernetes.io/projected/d4873c9c-f109-4fb1-9653-77b12a7bba1e-kube-api-access-dhcrp\") pod \"payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs\" (UID: \"d4873c9c-f109-4fb1-9653-77b12a7bba1e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" Oct 30 04:46:14.826830 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:14.826811 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" Oct 30 04:46:15.137538 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:15.137503 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs"] Oct 30 04:46:15.137757 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:46:15.137730 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4873c9c_f109_4fb1_9653_77b12a7bba1e.slice/crio-346e71bd586b378b7947b813c7030a2314d4a5ed0d4ae2b31c9008784bf5d684 WatchSource:0}: Error finding container 346e71bd586b378b7947b813c7030a2314d4a5ed0d4ae2b31c9008784bf5d684: Status 404 returned error can't find the container with id 346e71bd586b378b7947b813c7030a2314d4a5ed0d4ae2b31c9008784bf5d684 Oct 30 04:46:15.286501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:15.286459 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" event={"ID":"d4873c9c-f109-4fb1-9653-77b12a7bba1e","Type":"ContainerStarted","Data":"b1a10612c255c1db1be250770c5c7d1cd2f23eb6b270a68fda0826db565f99b5"} Oct 30 04:46:15.286705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:15.286517 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" event={"ID":"d4873c9c-f109-4fb1-9653-77b12a7bba1e","Type":"ContainerStarted","Data":"346e71bd586b378b7947b813c7030a2314d4a5ed0d4ae2b31c9008784bf5d684"} Oct 30 04:46:15.286705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:15.286634 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" Oct 30 04:46:15.296352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:15.296327 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:15.331489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:15.331405 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" podStartSLOduration=1.331392024 podStartE2EDuration="1.331392024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:15.327289035 +0000 UTC m=+2898.613163812" watchObservedRunningTime="2026-10-30 04:46:15.331392024 +0000 UTC m=+2898.617266800" Oct 30 04:46:16.662413 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:16.662371 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:16.985747 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:16.985709 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:18.010049 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.010004 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:18.321343 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.321266 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc"] Oct 30 04:46:18.324824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.324802 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.344753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.344551 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-91705f-openshift-default-dockercfg-6jv45\"" Oct 30 04:46:18.362193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.362165 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc"] Oct 30 04:46:18.449440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.441492 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-certs\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.449440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.441633 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-socket\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.449440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.441887 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-credential-socket\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.449440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.441934 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-podinfo\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.449440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.441982 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw5v6\" (UniqueName: \"kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-kube-api-access-lw5v6\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.449440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.442053 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-data\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.449440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.442073 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-token\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.449440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.442103 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3abc285-621f-4873-a77d-70d6f93a6c53-istiod-ca-cert\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.449440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.442127 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-envoy\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.545444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.541482 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:18.553324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.552030 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw5v6\" (UniqueName: \"kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-kube-api-access-lw5v6\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.553324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.552113 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-data\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.553324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.552136 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-token\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.553324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.552164 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3abc285-621f-4873-a77d-70d6f93a6c53-istiod-ca-cert\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.553324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.552188 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-envoy\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.553324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.552223 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-certs\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.553324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.552295 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-socket\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.553324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.552491 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-credential-socket\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.553324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.552516 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-podinfo\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.558396 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.557542 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3abc285-621f-4873-a77d-70d6f93a6c53-istiod-ca-cert\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.558396 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.558147 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-socket\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.561187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.558764 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-data\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.561187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.559005 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-credential-socket\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.561187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.559181 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-certs\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.595401 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.595320 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-envoy\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.611437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.605058 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw5v6\" (UniqueName: \"kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-kube-api-access-lw5v6\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.623442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.612707 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-podinfo\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.640503 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.640467 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-token\") pod \"e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.642154 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.641864 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:18.749765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.749642 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:18.971945 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:46:18.970901 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3abc285_621f_4873_a77d_70d6f93a6c53.slice/crio-16153143acfd253f3b26f85313ae9c0e14b8c4bcd6ffa35f6cac785695178889 WatchSource:0}: Error finding container 16153143acfd253f3b26f85313ae9c0e14b8c4bcd6ffa35f6cac785695178889: Status 404 returned error can't find the container with id 16153143acfd253f3b26f85313ae9c0e14b8c4bcd6ffa35f6cac785695178889 Oct 30 04:46:18.972168 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.972042 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc"] Oct 30 04:46:18.974516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.974474 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:46:18.974638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.974556 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:46:18.974638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:18.974599 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:46:19.322287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:19.322190 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" event={"ID":"b3abc285-621f-4873-a77d-70d6f93a6c53","Type":"ContainerStarted","Data":"3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950"} Oct 30 04:46:19.322287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:19.322263 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" event={"ID":"b3abc285-621f-4873-a77d-70d6f93a6c53","Type":"ContainerStarted","Data":"16153143acfd253f3b26f85313ae9c0e14b8c4bcd6ffa35f6cac785695178889"} Oct 30 04:46:19.371042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:19.370988 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" podStartSLOduration=1.370970844 podStartE2EDuration="1.370970844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:19.370434013 +0000 UTC m=+2902.656308800" watchObservedRunningTime="2026-10-30 04:46:19.370970844 +0000 UTC m=+2902.656845620" Oct 30 04:46:19.643256 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:19.643185 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:20.341954 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:20.341921 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:20.649410 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:20.649341 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:21.336191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:21.336164 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:21.337085 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:21.337062 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:46:21.826742 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:21.826705 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:22.051555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:22.051520 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:22.275780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:22.275735 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:23.537754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:23.537713 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:23.778467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:23.766292 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:24.078916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:24.078873 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:25.788287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:25.788253 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:26.297457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:26.297408 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" Oct 30 04:46:27.531353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:27.531319 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:29.211034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:29.211000 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:30.810902 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:30.810869 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:32.439774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:32.439737 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:34.052496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:34.052460 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:35.824512 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:35.824461 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:37.513872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:37.513838 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:37.938487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:37.938450 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:39.199228 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:39.199192 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:40.683461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.683408 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p"] Oct 30 04:46:40.687960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.687938 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:40.710888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.710860 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-503804-serving-cert\"" Oct 30 04:46:40.713109 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.713085 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p"] Oct 30 04:46:40.779147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.779115 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwwk9\" (UniqueName: \"kubernetes.io/projected/757c4374-aa95-499b-b5e4-32c40a00ec1f-kube-api-access-vwwk9\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:40.779321 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.779158 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-metrics-certs\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:40.779456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.779397 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757c4374-aa95-499b-b5e4-32c40a00ec1f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:40.779569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.779554 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:40.889249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.889207 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:40.890118 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.890085 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwwk9\" (UniqueName: \"kubernetes.io/projected/757c4374-aa95-499b-b5e4-32c40a00ec1f-kube-api-access-vwwk9\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:40.890307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.890147 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-metrics-certs\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:40.890376 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.890363 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757c4374-aa95-499b-b5e4-32c40a00ec1f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:40.890499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.890487 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:40.891106 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:46:40.891084 4793 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-serving-cert: secret "maas-api-e2e-extmod-b-503804-serving-cert" not found Oct 30 04:46:40.891323 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:46:40.891296 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-maas-api-tls podName:757c4374-aa95-499b-b5e4-32c40a00ec1f nodeName:}" failed. No retries permitted until 2026-10-30 04:46:41.391275369 +0000 UTC m=+2924.677150122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-maas-api-tls") pod "maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" (UID: "757c4374-aa95-499b-b5e4-32c40a00ec1f") : secret "maas-api-e2e-extmod-b-503804-serving-cert" not found Oct 30 04:46:40.891515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.891430 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757c4374-aa95-499b-b5e4-32c40a00ec1f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:40.894128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.894101 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-metrics-certs\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:40.917013 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:40.916987 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwwk9\" (UniqueName: \"kubernetes.io/projected/757c4374-aa95-499b-b5e4-32c40a00ec1f-kube-api-access-vwwk9\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:41.098301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:41.098230 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:41.219206 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:46:41.219169 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:46:41.397851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:41.397764 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:41.400432 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:41.400398 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:41.608257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:41.608212 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:41.917598 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:46:41.917568 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757c4374_aa95_499b_b5e4_32c40a00ec1f.slice/crio-931af0126317af8349f5a43626967a022984be8a63d3aa4484e71e78a2a0f422 WatchSource:0}: Error finding container 931af0126317af8349f5a43626967a022984be8a63d3aa4484e71e78a2a0f422: Status 404 returned error can't find the container with id 931af0126317af8349f5a43626967a022984be8a63d3aa4484e71e78a2a0f422 Oct 30 04:46:41.919215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:41.919182 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p"] Oct 30 04:46:42.520922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:42.520823 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" event={"ID":"757c4374-aa95-499b-b5e4-32c40a00ec1f","Type":"ContainerStarted","Data":"0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2"} Oct 30 04:46:42.520922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:42.520902 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" event={"ID":"757c4374-aa95-499b-b5e4-32c40a00ec1f","Type":"ContainerStarted","Data":"931af0126317af8349f5a43626967a022984be8a63d3aa4484e71e78a2a0f422"} Oct 30 04:46:42.521164 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:42.520971 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:42.566907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:42.566856 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" podStartSLOduration=2.221255368 podStartE2EDuration="2.566842702s" totalImagesPullingTime="345.587334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:42.563012114 +0000 UTC m=+2925.848886892" watchObservedRunningTime="2026-10-30 04:46:42.566842702 +0000 UTC m=+2925.852717493" Oct 30 04:46:42.600985 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:42.600953 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:43.425799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:43.425761 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:44.129253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:44.129214 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:45.879440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:45.878925 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:46.093050 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:46.093010 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:46.307199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:46.307161 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:47.553002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:47.552964 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:47.778689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:47.778653 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:48.118818 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:48.118775 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:48.532952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:48.532924 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 04:46:48.711077 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:48.696913 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:48.947098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:48.947063 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:49.547759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:49.547727 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:51.351553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:51.351514 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:52.864967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:52.864932 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:53.231782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:53.231647 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:53.473178 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:53.473132 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:54.728662 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:54.728624 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:54.974245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:54.974210 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:55.190829 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:55.190798 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:55.923705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:55.923672 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:57.017763 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:57.017722 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:58.667484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.667440 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl"] Oct 30 04:46:58.670782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.670766 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.700024 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.699997 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-91705f-serving-cert\"" Oct 30 04:46:58.706622 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.706590 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl"] Oct 30 04:46:58.793623 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.793591 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.793825 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.793807 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-metrics-certs\") pod \"maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.793929 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.793911 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a910f96e-a20b-40a7-9d33-421b71a81f7b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.793980 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.793965 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkmgh\" (UniqueName: \"kubernetes.io/projected/a910f96e-a20b-40a7-9d33-421b71a81f7b-kube-api-access-hkmgh\") pod \"maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.871816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.871775 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:58.897445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.896691 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.897445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.896912 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-metrics-certs\") pod \"maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.897445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.896985 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a910f96e-a20b-40a7-9d33-421b71a81f7b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.897445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.897043 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkmgh\" (UniqueName: \"kubernetes.io/projected/a910f96e-a20b-40a7-9d33-421b71a81f7b-kube-api-access-hkmgh\") pod \"maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.912884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.904982 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.913232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.911384 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a910f96e-a20b-40a7-9d33-421b71a81f7b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.926240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.926197 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-metrics-certs\") pod \"maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.928129 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.928106 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkmgh\" (UniqueName: \"kubernetes.io/projected/a910f96e-a20b-40a7-9d33-421b71a81f7b-kube-api-access-hkmgh\") pod \"maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:58.988005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:58.987983 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:59.107306 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:59.107271 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:46:59.304773 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:46:59.304742 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda910f96e_a20b_40a7_9d33_421b71a81f7b.slice/crio-4a170c795f668b3f7842782b5fc2e72b70394b36de4265cfb7cf6f6ff6d0fce6 WatchSource:0}: Error finding container 4a170c795f668b3f7842782b5fc2e72b70394b36de4265cfb7cf6f6ff6d0fce6: Status 404 returned error can't find the container with id 4a170c795f668b3f7842782b5fc2e72b70394b36de4265cfb7cf6f6ff6d0fce6 Oct 30 04:46:59.305342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:59.305317 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl"] Oct 30 04:46:59.786800 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:59.786768 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" event={"ID":"a910f96e-a20b-40a7-9d33-421b71a81f7b","Type":"ContainerStarted","Data":"a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0"} Oct 30 04:46:59.787570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:59.786822 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" event={"ID":"a910f96e-a20b-40a7-9d33-421b71a81f7b","Type":"ContainerStarted","Data":"4a170c795f668b3f7842782b5fc2e72b70394b36de4265cfb7cf6f6ff6d0fce6"} Oct 30 04:46:59.787570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:59.786843 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:46:59.830764 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:46:59.830718 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" podStartSLOduration=1.522287932 podStartE2EDuration="1.830704745s" totalImagesPullingTime="308.416813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:46:59.828718238 +0000 UTC m=+2943.114593014" watchObservedRunningTime="2026-10-30 04:46:59.830704745 +0000 UTC m=+2943.116579555" Oct 30 04:47:00.724055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:00.724019 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:02.418034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:02.417998 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:04.126194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:04.126157 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:05.797224 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:05.797183 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:05.801740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:05.801720 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 04:47:07.514028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:07.513993 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:09.294156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:09.294123 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:11.213853 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:11.213810 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:11.220037 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:47:11.220012 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:47:12.712066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:12.712029 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:14.526110 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:14.526080 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:16.215216 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:16.215181 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:17.824225 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:17.824177 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:19.505562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:19.505520 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:21.533342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:21.533301 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:22.441844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:22.441806 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:22.680298 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:22.680260 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:23.238625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:23.238581 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:23.472466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:23.470001 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:23.690924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:23.690890 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:24.241444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:24.241384 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:25.319197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:25.319151 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:25.536271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:25.536231 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:25.756166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:25.756125 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:25.988305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:25.988272 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:26.221586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:26.221548 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:26.783808 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:26.783769 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:27.011231 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:27.011194 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:27.331632 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:27.331597 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:28.871378 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:28.871274 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:29.123630 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:29.123554 4793 generic.go:356] "Generic (PLEG): container finished" podID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerID="3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950" exitCode=0 Oct 30 04:47:29.123630 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:29.123599 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" event={"ID":"b3abc285-621f-4873-a77d-70d6f93a6c53","Type":"ContainerDied","Data":"3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950"} Oct 30 04:47:29.124140 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:29.124123 4793 scope.go:122] "RemoveContainer" containerID="3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950" Oct 30 04:47:29.126068 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:29.126038 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:47:29.126156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:29.126114 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:47:29.126156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:29.126149 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:47:29.220161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:29.220130 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:29.467356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:29.467320 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:30.132603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:30.132566 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" event={"ID":"b3abc285-621f-4873-a77d-70d6f93a6c53","Type":"ContainerStarted","Data":"442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02"} Oct 30 04:47:30.434261 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:30.434223 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:30.643005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:30.642966 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:47:30.643188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:30.643133 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 04:47:30.643237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:30.643191 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 04:47:30.690526 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:30.686336 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:30.918272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:30.918224 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:31.165089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:31.164956 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:31.512685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:31.512649 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:31.642922 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:31.642883 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 04:47:31.643093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:31.642932 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 04:47:31.813182 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:31.813094 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:32.022844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.022810 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:32.593000 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.592965 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7"] Oct 30 04:47:32.596618 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.596598 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.618278 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.618114 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7"] Oct 30 04:47:32.621599 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.621557 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-721c85-serving-cert\"" Oct 30 04:47:32.643030 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.643001 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 04:47:32.643533 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.643490 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 04:47:32.644104 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.644077 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-maas-api-tls\") pod \"maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.644218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.644126 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mswn9\" (UniqueName: \"kubernetes.io/projected/a6c23d07-4050-438b-951a-03f5d882fbef-kube-api-access-mswn9\") pod \"maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.644218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.644204 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-metrics-certs\") pod \"maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.644370 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.644350 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c23d07-4050-438b-951a-03f5d882fbef-openshift-service-ca\") pod \"maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.745383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.745300 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-maas-api-tls\") pod \"maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.745383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.745336 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mswn9\" (UniqueName: \"kubernetes.io/projected/a6c23d07-4050-438b-951a-03f5d882fbef-kube-api-access-mswn9\") pod \"maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.745383 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.745380 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-metrics-certs\") pod \"maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.745756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.745476 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c23d07-4050-438b-951a-03f5d882fbef-openshift-service-ca\") pod \"maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.746141 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.746121 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c23d07-4050-438b-951a-03f5d882fbef-openshift-service-ca\") pod \"maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.747934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.747912 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-metrics-certs\") pod \"maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.748045 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.747944 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-maas-api-tls\") pod \"maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.785846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.774240 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mswn9\" (UniqueName: \"kubernetes.io/projected/a6c23d07-4050-438b-951a-03f5d882fbef-kube-api-access-mswn9\") pod \"maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:32.789437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.788964 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:32.911811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:32.911777 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:33.006351 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:33.006144 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:33.285723 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:47:33.285692 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c23d07_4050_438b_951a_03f5d882fbef.slice/crio-3eabeb09c711635d2314df6a403c94182c62450a159f78478635f937e4f69434 WatchSource:0}: Error finding container 3eabeb09c711635d2314df6a403c94182c62450a159f78478635f937e4f69434: Status 404 returned error can't find the container with id 3eabeb09c711635d2314df6a403c94182c62450a159f78478635f937e4f69434 Oct 30 04:47:33.291304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:33.291277 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7"] Oct 30 04:47:33.643410 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:33.643383 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 04:47:33.643768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:33.643451 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 04:47:34.024112 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:34.024081 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:34.226438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:34.226380 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" event={"ID":"a6c23d07-4050-438b-951a-03f5d882fbef","Type":"ContainerStarted","Data":"f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c"} Oct 30 04:47:34.226663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:34.226460 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:34.226663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:34.226481 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" event={"ID":"a6c23d07-4050-438b-951a-03f5d882fbef","Type":"ContainerStarted","Data":"3eabeb09c711635d2314df6a403c94182c62450a159f78478635f937e4f69434"} Oct 30 04:47:34.281282 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:34.281160 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" podStartSLOduration=1.966026788 podStartE2EDuration="2.281141963s" totalImagesPullingTime="315.115175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:47:34.278645964 +0000 UTC m=+2977.564520740" watchObservedRunningTime="2026-10-30 04:47:34.281141963 +0000 UTC m=+2977.567016739" Oct 30 04:47:34.608249 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:34.608167 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:34.643214 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:34.643186 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 04:47:34.643362 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:34.643234 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 04:47:35.118584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:35.118550 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:35.343832 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:35.343804 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:35.586556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:35.576373 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:35.652560 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:35.652520 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" start-of-body= Oct 30 04:47:35.653346 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:35.653300 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.204:15021/healthz/ready\": dial tcp 10.133.0.204:15021: connect: connection refused" Oct 30 04:47:35.789976 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:35.789938 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:36.301624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:36.301588 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:36.336357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:36.336329 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:47:36.647642 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:36.647572 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:47:37.340497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:37.340472 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 04:47:37.925220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:37.925188 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:39.715223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:39.715179 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:40.332235 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:40.332205 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:47:41.224174 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:47:41.224132 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:47:41.406990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:41.406952 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:42.594459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:42.594410 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:42.813255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:42.813221 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:43.295354 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:43.295315 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:43.545112 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:43.545074 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:44.726073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:44.726045 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:44.973077 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:44.973042 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:45.220530 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:45.220495 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:45.468803 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:45.468765 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:45.787546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:45.787510 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:46.102519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:46.102398 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:46.390497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:46.386968 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:47.335819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:47.335789 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:47.581057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:47.581021 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:49.292010 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:49.291968 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:50.146016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:50.145977 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:50.362650 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:50.362615 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:50.680952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:50.680905 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:51.009370 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:51.009283 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:51.242143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:51.242106 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:51.767961 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:51.767922 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:52.978828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:52.978797 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:54.633149 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:54.633112 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:56.021210 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:56.021174 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:56.340648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:56.340559 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:58.127213 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:58.127171 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:47:59.811115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:47:59.811077 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:01.527010 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:01.526972 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:03.307320 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:03.307267 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:05.238222 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:05.238180 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:06.837679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:06.837642 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:08.545532 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:08.545490 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:10.230689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:10.230601 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:11.219879 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:48:11.219845 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:48:11.910082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:11.910049 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:13.592272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:13.592233 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:15.308946 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:15.308910 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:17.037674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:17.037640 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:18.902174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:18.902141 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:19.691742 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:19.691707 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:20.704268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:20.704236 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:22.277323 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:22.277288 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:22.550247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:22.550164 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:24.050591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:24.050544 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:25.799259 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:25.799218 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:27.835798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:27.835762 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:29.198949 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:29.198918 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:30.795115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:30.795075 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:32.499246 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:32.499207 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:34.395254 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:34.394015 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:36.128327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:36.128287 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:37.828921 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:37.828882 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:38.093544 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:38.093413 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:38.757712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:38.757674 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:38.975389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:38.975352 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:39.782088 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:39.782052 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:41.218939 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:48:41.218903 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:48:41.515260 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:41.515164 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:43.370125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:43.370088 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:45.034312 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:45.034281 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:46.722016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:46.721982 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:47.799185 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:47.799144 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:48.271926 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.271891 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:48.658908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.658821 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:48.849054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.849023 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz"] Oct 30 04:48:48.853717 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.853693 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:48.875055 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.875022 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz"] Oct 30 04:48:48.882633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.882609 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-721c85-openshift-default-dockercfg-62hls\"" Oct 30 04:48:48.955310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.954995 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-data\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:48.955536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.955323 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istiod-ca-cert\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:48.955536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.955500 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-podinfo\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:48.955674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.955537 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-envoy\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:48.955674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.955628 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv2f7\" (UniqueName: \"kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-kube-api-access-qv2f7\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:48.955813 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.955706 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-credential-socket\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:48.955912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.955890 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-socket\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:48.956047 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.956027 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-token\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:48.956128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:48.956109 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-certs\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.057793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.057761 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-data\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.057793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.057796 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istiod-ca-cert\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.058035 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.057850 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-podinfo\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.058035 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.057867 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-envoy\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.058035 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.057888 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv2f7\" (UniqueName: \"kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-kube-api-access-qv2f7\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.058035 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.057916 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-credential-socket\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.058035 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.057944 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-socket\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.058325 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.058228 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-data\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.058325 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.058234 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-token\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.058325 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.058270 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-socket\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.058516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.058354 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-certs\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.058676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.058651 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-credential-socket\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.058867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.058849 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-certs\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.058960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.058942 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istiod-ca-cert\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.060619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.060592 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-envoy\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.060748 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.060718 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-podinfo\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.109928 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.109902 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-token\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.110041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.109947 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv2f7\" (UniqueName: \"kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-kube-api-access-qv2f7\") pod \"e2e-autores-721c85-openshift-default-7dfccc8995-f54hz\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.168456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.168435 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:49.496071 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.496040 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz"] Oct 30 04:48:49.496846 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:48:49.496820 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a4adc7_e763_4c00_bb3e_8ef66c40a1ab.slice/crio-4ecb9cc01cd5a5875f85d92722e9eef983ed1c2ccd26f0a65f9fd661da8d759c WatchSource:0}: Error finding container 4ecb9cc01cd5a5875f85d92722e9eef983ed1c2ccd26f0a65f9fd661da8d759c: Status 404 returned error can't find the container with id 4ecb9cc01cd5a5875f85d92722e9eef983ed1c2ccd26f0a65f9fd661da8d759c Oct 30 04:48:49.500192 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.500153 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:48:49.500304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.500225 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:48:49.500304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.500255 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:48:49.997086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.997041 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" event={"ID":"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab","Type":"ContainerStarted","Data":"f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42"} Oct 30 04:48:49.997656 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:49.997107 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" event={"ID":"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab","Type":"ContainerStarted","Data":"4ecb9cc01cd5a5875f85d92722e9eef983ed1c2ccd26f0a65f9fd661da8d759c"} Oct 30 04:48:50.048525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:50.048405 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" podStartSLOduration=2.04838299 podStartE2EDuration="2.04838299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:48:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:48:50.045552812 +0000 UTC m=+3053.331427588" watchObservedRunningTime="2026-10-30 04:48:50.04838299 +0000 UTC m=+3053.334257773" Oct 30 04:48:50.170439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:50.169797 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:50.175873 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:50.175538 4793 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 04:48:50.175873 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:50.175598 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" podUID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 04:48:50.419426 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:50.419322 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:51.169759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:51.169720 4793 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 04:48:51.170360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:51.170329 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" podUID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 04:48:52.169441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:52.169220 4793 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 04:48:52.169441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:52.169282 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" podUID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 04:48:52.226982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:52.226948 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:53.170757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:53.170719 4793 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 04:48:53.171338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:53.171308 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" podUID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 04:48:53.272479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:53.272444 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:53.928805 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:53.928768 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:54.171150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:54.170695 4793 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 04:48:54.171150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:54.170784 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" podUID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 04:48:55.172179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:55.172143 4793 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 04:48:55.172824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:55.172794 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" podUID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 04:48:55.694912 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:55.694876 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:56.137982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:56.137898 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:56.175032 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:56.174999 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:56.175519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:56.175413 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:56.176048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:56.176027 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:48:57.407783 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:57.407751 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:48:59.032567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:48:59.032524 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:00.922465 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:00.922396 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:02.776188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:02.776143 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:03.117761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:03.116919 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:03.343334 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:03.343295 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:03.807859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:03.807824 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:04.044440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:04.037986 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:04.560602 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:04.560569 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:04.920110 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:04.920013 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:05.233403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:05.233369 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:06.161363 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:06.161327 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:06.686120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:06.686082 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:08.710204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:08.710172 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:10.178682 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:10.178633 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:10.400729 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:10.400697 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:10.635808 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:10.635735 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:10.859076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:10.859027 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:11.368333 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:49:11.368290 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:49:11.822225 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:11.822183 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:12.379932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:12.379897 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:12.693403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:12.693369 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:14.266719 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:14.266683 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:16.040589 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:16.040552 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:17.725611 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:17.725572 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:19.507157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:19.507117 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:21.212209 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:21.212166 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:22.272563 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:22.272526 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:22.511359 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:22.511267 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:22.944758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:22.944718 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:23.167313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:23.167278 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:24.709248 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:24.709208 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:24.938885 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:24.938846 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:25.210440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:25.201344 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:25.800129 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:25.800091 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:26.038553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:26.038516 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:26.256064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:26.256013 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:26.585648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:26.585551 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:26.846327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:26.846202 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:27.099207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:27.099124 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:27.342678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:27.342639 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:27.836110 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:27.836069 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:28.084439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:28.083147 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:28.714831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:28.714799 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:28.937956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:28.937919 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:29.344800 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:29.344761 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:29.564722 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:29.564686 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:30.850221 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:30.850182 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:31.078571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:31.078502 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:31.300976 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:31.300932 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:32.085388 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:32.085350 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:32.912789 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:32.912702 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:34.705162 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:34.705120 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:36.032480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.032449 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7"] Oct 30 04:49:36.032884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.032681 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" podUID="a6c23d07-4050-438b-951a-03f5d882fbef" containerName="maas-api" containerID="cri-o://f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c" gracePeriod=30 Oct 30 04:49:36.183252 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.183213 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:36.428080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.428050 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:49:36.592591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.592554 4793 generic.go:356] "Generic (PLEG): container finished" podID="a6c23d07-4050-438b-951a-03f5d882fbef" containerID="f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c" exitCode=0 Oct 30 04:49:36.593047 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.593002 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" event={"ID":"a6c23d07-4050-438b-951a-03f5d882fbef","Type":"ContainerDied","Data":"f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c"} Oct 30 04:49:36.593225 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.593205 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" event={"ID":"a6c23d07-4050-438b-951a-03f5d882fbef","Type":"ContainerDied","Data":"3eabeb09c711635d2314df6a403c94182c62450a159f78478635f937e4f69434"} Oct 30 04:49:36.593380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.593363 4793 scope.go:122] "RemoveContainer" containerID="f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c" Oct 30 04:49:36.593700 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.593680 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7" Oct 30 04:49:36.618350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.617298 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-metrics-certs\") pod \"a6c23d07-4050-438b-951a-03f5d882fbef\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " Oct 30 04:49:36.618350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.617392 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-maas-api-tls\") pod \"a6c23d07-4050-438b-951a-03f5d882fbef\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " Oct 30 04:49:36.618350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.617448 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6c23d07-4050-438b-951a-03f5d882fbef-kube-api-access-mswn9\" (UniqueName: \"kubernetes.io/projected/a6c23d07-4050-438b-951a-03f5d882fbef-kube-api-access-mswn9\") pod \"a6c23d07-4050-438b-951a-03f5d882fbef\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " Oct 30 04:49:36.618350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.617530 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6c23d07-4050-438b-951a-03f5d882fbef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c23d07-4050-438b-951a-03f5d882fbef-openshift-service-ca\") pod \"a6c23d07-4050-438b-951a-03f5d882fbef\" (UID: \"a6c23d07-4050-438b-951a-03f5d882fbef\") " Oct 30 04:49:36.618350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.618119 4793 scope.go:122] "RemoveContainer" containerID="f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c" Oct 30 04:49:36.619971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.619230 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6c23d07-4050-438b-951a-03f5d882fbef-openshift-service-ca" pod "a6c23d07-4050-438b-951a-03f5d882fbef" (UID: "a6c23d07-4050-438b-951a-03f5d882fbef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:49:36.619971 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:49:36.619350 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c\": container with ID starting with f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c not found: ID does not exist" containerID="f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c" Oct 30 04:49:36.620679 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.619386 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c"} err="failed to get container status \"f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c\": rpc error: code = NotFound desc = could not find container \"f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c\": container with ID starting with f949bc0e433f12c2ecca94543dcf3402c0f883d195fa84a5beb80e5452cd683c not found: ID does not exist" Oct 30 04:49:36.624649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.624602 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-metrics-certs" pod "a6c23d07-4050-438b-951a-03f5d882fbef" (UID: "a6c23d07-4050-438b-951a-03f5d882fbef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:49:36.624849 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.624823 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-maas-api-tls" pod "a6c23d07-4050-438b-951a-03f5d882fbef" (UID: "a6c23d07-4050-438b-951a-03f5d882fbef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 04:49:36.629618 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.629584 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c23d07-4050-438b-951a-03f5d882fbef-kube-api-access-mswn9" pod "a6c23d07-4050-438b-951a-03f5d882fbef" (UID: "a6c23d07-4050-438b-951a-03f5d882fbef"). InnerVolumeSpecName "kube-api-access-mswn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:49:36.719643 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.719607 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:36.719643 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.719644 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6c23d07-4050-438b-951a-03f5d882fbef-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:36.719913 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.719671 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mswn9\" (UniqueName: \"kubernetes.io/projected/a6c23d07-4050-438b-951a-03f5d882fbef-kube-api-access-mswn9\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:36.719913 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.719688 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6c23d07-4050-438b-951a-03f5d882fbef-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:36.815382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.815350 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:36.941437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.941386 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7"] Oct 30 04:49:36.947996 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:36.947967 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-721c85-675c9cd7f4-jzgv7"] Oct 30 04:49:37.053737 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:37.053661 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:37.520867 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:37.520832 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:38.385727 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:38.385694 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:38.622258 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:38.622226 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:38.849034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:38.849003 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6c23d07-4050-438b-951a-03f5d882fbef" path="/var/lib/kubelet/pods/a6c23d07-4050-438b-951a-03f5d882fbef/volumes" Oct 30 04:49:38.873965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:38.873926 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:39.156982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:39.156879 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:39.371018 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:39.370981 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:39.750651 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:39.750574 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:39.973670 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:39.973636 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:40.220051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:40.220015 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:40.576004 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:40.575908 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:40.795473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:40.795394 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:40.879595 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:40.879518 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz"] Oct 30 04:49:40.881021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:40.880982 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" podUID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" containerName="istio-proxy" containerID="cri-o://f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42" gracePeriod=30 Oct 30 04:49:41.041592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:41.041557 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:41.177178 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:41.177142 4793 patch_prober.go:29] interesting pod/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 04:49:41.177345 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:41.177202 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" podUID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 04:49:41.220741 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:49:41.220705 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:49:41.271100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:41.271064 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:41.509328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:41.509214 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:42.940884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:42.940841 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:43.159622 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:43.159580 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:43.549491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:43.549443 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:43.877622 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:43.877536 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:44.643729 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:44.643695 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:44.918231 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:44.918143 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:46.195453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.195392 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:49:46.245325 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.245298 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-podinfo\") pod \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " Oct 30 04:49:46.245516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.245366 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-kube-api-access-qv2f7\" (UniqueName: \"kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-kube-api-access-qv2f7\") pod \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " Oct 30 04:49:46.245516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.245451 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istiod-ca-cert\") pod \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " Oct 30 04:49:46.245516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.245496 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-token\" (UniqueName: \"kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-token\") pod \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " Oct 30 04:49:46.245705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.245540 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-socket\") pod \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " Oct 30 04:49:46.245705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.245580 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-certs\") pod \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " Oct 30 04:49:46.245705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.245624 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-data\") pod \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " Oct 30 04:49:46.245887 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.245727 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-envoy\") pod \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " Oct 30 04:49:46.245950 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.245892 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-credential-socket\") pod \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\" (UID: \"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab\") " Oct 30 04:49:46.246054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.246011 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-certs" pod "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" (UID: "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:49:46.246515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.246240 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-credential-socket" pod "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" (UID: "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:49:46.246633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.246587 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:46.246633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.246612 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:46.246633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.246622 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-socket" pod "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" (UID: "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:49:46.246892 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.246862 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-data" pod "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" (UID: "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:49:46.247023 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.246929 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istiod-ca-cert" pod "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" (UID: "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:49:46.247811 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.247779 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-podinfo" pod "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" (UID: "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:49:46.248622 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.248594 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-kube-api-access-qv2f7" pod "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" (UID: "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab"). InnerVolumeSpecName "kube-api-access-qv2f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:49:46.249492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.249465 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-token" pod "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" (UID: "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:49:46.249847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.249829 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-envoy" pod "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" (UID: "18a4adc7-e763-4c00-bb3e-8ef66c40a1ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:49:46.348115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.348081 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv2f7\" (UniqueName: \"kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-kube-api-access-qv2f7\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:46.348115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.348111 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:46.348115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.348120 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:46.348333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.348128 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:46.348333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.348137 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:46.348333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.348146 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:46.348333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.348154 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:49:46.706861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.706824 4793 generic.go:356] "Generic (PLEG): container finished" podID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" containerID="f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42" exitCode=0 Oct 30 04:49:46.707066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.706881 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" event={"ID":"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab","Type":"ContainerDied","Data":"f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42"} Oct 30 04:49:46.707066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.706891 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" Oct 30 04:49:46.707066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.706937 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz" event={"ID":"18a4adc7-e763-4c00-bb3e-8ef66c40a1ab","Type":"ContainerDied","Data":"4ecb9cc01cd5a5875f85d92722e9eef983ed1c2ccd26f0a65f9fd661da8d759c"} Oct 30 04:49:46.707066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.706970 4793 scope.go:122] "RemoveContainer" containerID="f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42" Oct 30 04:49:46.709475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.709412 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:46.721744 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.721717 4793 scope.go:122] "RemoveContainer" containerID="f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42" Oct 30 04:49:46.722091 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:49:46.722069 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42\": container with ID starting with f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42 not found: ID does not exist" containerID="f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42" Oct 30 04:49:46.722172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.722098 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42"} err="failed to get container status \"f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42\": rpc error: code = NotFound desc = could not find container \"f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42\": container with ID starting with f3a2731e5e1ac278d4ffbabfdb8308f3bb6b1a0999bacd78bba484215e5c8c42 not found: ID does not exist" Oct 30 04:49:46.761382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.761353 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz"] Oct 30 04:49:46.773964 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.773936 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-721c85-openshift-default-7dfccc8995-f54hz"] Oct 30 04:49:46.847880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:46.847849 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" path="/var/lib/kubelet/pods/18a4adc7-e763-4c00-bb3e-8ef66c40a1ab/volumes" Oct 30 04:49:47.487999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:47.487959 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:47.743934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:47.743828 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:48.516667 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:48.516632 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:49.152820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.152784 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:49.420495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.412249 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx"] Oct 30 04:49:49.420495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.415248 4793 cpu_manager.go:414] "containerMap: removing container" podUID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" containerName="istio-proxy" Oct 30 04:49:49.420495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.415274 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" containerName="istio-proxy" Oct 30 04:49:49.420495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.415315 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a6c23d07-4050-438b-951a-03f5d882fbef" containerName="maas-api" Oct 30 04:49:49.420495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.415324 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6c23d07-4050-438b-951a-03f5d882fbef" containerName="maas-api" Oct 30 04:49:49.420495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.415620 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a4adc7-e763-4c00-bb3e-8ef66c40a1ab" containerName="istio-proxy" Oct 30 04:49:49.420495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.415649 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6c23d07-4050-438b-951a-03f5d882fbef" containerName="maas-api" Oct 30 04:49:49.443440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.422817 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" Oct 30 04:49:49.443440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.443307 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c0d6e7\"" Oct 30 04:49:49.449559 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.443477 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c0d6e7-dockercfg-xxc8s\"" Oct 30 04:49:49.479662 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.479626 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx"] Oct 30 04:49:49.488971 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.488743 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:49.587154 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.587113 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbcgr\" (UniqueName: \"kubernetes.io/projected/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-kube-api-access-cbcgr\") pod \"payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx\" (UID: \"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" Oct 30 04:49:49.587579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.587396 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx\" (UID: \"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" Oct 30 04:49:49.688884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.688845 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbcgr\" (UniqueName: \"kubernetes.io/projected/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-kube-api-access-cbcgr\") pod \"payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx\" (UID: \"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" Oct 30 04:49:49.689056 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.689040 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx\" (UID: \"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" Oct 30 04:49:49.690020 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.689995 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx\" (UID: \"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" Oct 30 04:49:49.713864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.713833 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbcgr\" (UniqueName: \"kubernetes.io/projected/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-kube-api-access-cbcgr\") pod \"payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx\" (UID: \"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" Oct 30 04:49:49.748376 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:49.748350 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" Oct 30 04:49:50.061443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:50.061400 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx"] Oct 30 04:49:50.062216 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:49:50.062187 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a9fa6b_a5b2_407f_87c3_bebc0e0922df.slice/crio-f2f890a0639bdf23b2e9d9d0d28454eb21e9349d9f1aeef3cf74f810169c624d WatchSource:0}: Error finding container f2f890a0639bdf23b2e9d9d0d28454eb21e9349d9f1aeef3cf74f810169c624d: Status 404 returned error can't find the container with id f2f890a0639bdf23b2e9d9d0d28454eb21e9349d9f1aeef3cf74f810169c624d Oct 30 04:49:50.300343 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:50.300298 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:50.739318 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:50.739280 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" event={"ID":"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df","Type":"ContainerStarted","Data":"da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307"} Oct 30 04:49:50.739740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:50.739331 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" event={"ID":"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df","Type":"ContainerStarted","Data":"f2f890a0639bdf23b2e9d9d0d28454eb21e9349d9f1aeef3cf74f810169c624d"} Oct 30 04:49:50.739740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:50.739499 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" Oct 30 04:49:50.786122 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:50.786073 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" podStartSLOduration=1.786061781 podStartE2EDuration="1.786061781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:50.784797782 +0000 UTC m=+3114.070672576" watchObservedRunningTime="2026-10-30 04:49:50.786061781 +0000 UTC m=+3114.071936556" Oct 30 04:49:51.245012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:51.244975 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:51.457649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:51.457613 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:51.700877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:51.700847 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:51.913529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:51.913497 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:52.137398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:52.137313 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:52.351569 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:52.351537 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:52.778075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:52.778043 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:53.039525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.039430 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:53.580381 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.580339 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:53.751804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.751769 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd"] Oct 30 04:49:53.756526 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.756498 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:53.792535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.792500 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"payload-processing-external-model-e2e--545d7aee-dockercfg-rcsxv\"" Oct 30 04:49:53.812172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.812146 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd"] Oct 30 04:49:53.820044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.820024 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"openshift-service-ca.crt\"" Oct 30 04:49:53.820322 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.820229 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"routing-overlay\"" Oct 30 04:49:53.820322 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.820286 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"payload-processing-external-model-plugins-e2e-extmod-a-05bee8\"" Oct 30 04:49:53.820530 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.820402 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"kube-root-ca.crt\"" Oct 30 04:49:53.891754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.869399 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:53.906220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.895567 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:53.925673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.925610 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4pr9\" (UniqueName: \"kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-kube-api-access-m4pr9\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:53.951452 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.938729 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:53.964597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.891752 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-05bee8\"/\"agc-cred-b517e77fd85b\"" Oct 30 04:49:53.990792 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:53.990742 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:54.066277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.066243 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:54.066799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.066313 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:54.066799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.066446 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4pr9\" (UniqueName: \"kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-kube-api-access-m4pr9\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:54.066799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.066532 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:54.066988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.066969 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:54.067488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.067462 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:54.068812 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.068785 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:54.089673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.089645 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4pr9\" (UniqueName: \"kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-kube-api-access-m4pr9\") pod \"payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:54.370917 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.370881 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:54.680807 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:49:54.680773 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0e8b2e_bf65_4209_9358_7f69e37e4e28.slice/crio-3994dbfee68a5f7d16c1b29d8ebbc6ed2b342e67748f1f5b2d59c752c794d7f5 WatchSource:0}: Error finding container 3994dbfee68a5f7d16c1b29d8ebbc6ed2b342e67748f1f5b2d59c752c794d7f5: Status 404 returned error can't find the container with id 3994dbfee68a5f7d16c1b29d8ebbc6ed2b342e67748f1f5b2d59c752c794d7f5 Oct 30 04:49:54.682519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.682490 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd"] Oct 30 04:49:54.775596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.775564 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" event={"ID":"7b0e8b2e-bf65-4209-9358-7f69e37e4e28","Type":"ContainerStarted","Data":"92d6b41af503b89b04fed8af7fca50661e3cba2d395dd98b63919abd9217b209"} Oct 30 04:49:54.775716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.775626 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" event={"ID":"7b0e8b2e-bf65-4209-9358-7f69e37e4e28","Type":"ContainerStarted","Data":"3994dbfee68a5f7d16c1b29d8ebbc6ed2b342e67748f1f5b2d59c752c794d7f5"} Oct 30 04:49:54.775716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.775653 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:49:54.819721 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:54.819671 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" podStartSLOduration=1.819658242 podStartE2EDuration="1.819658242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:49:54.818175893 +0000 UTC m=+3118.104050680" watchObservedRunningTime="2026-10-30 04:49:54.819658242 +0000 UTC m=+3118.105533017" Oct 30 04:49:55.596170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:55.596132 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:56.279551 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:56.279509 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:57.222925 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:57.222890 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:49:58.900359 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:49:58.900312 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:00.578113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:00.578066 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:01.750691 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:01.750664 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" Oct 30 04:50:04.268183 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:04.268145 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:04.489150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:04.489117 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:05.640835 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:05.640799 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:05.785287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:05.785262 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:50:07.433445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:07.433379 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:09.139566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:09.139531 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:10.877623 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:10.877577 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:11.219551 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:50:11.219516 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:50:12.680014 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:12.679975 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:14.347944 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:14.347905 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:16.104211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:16.104174 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:17.908966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:17.908935 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:19.697351 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:19.697321 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:21.222112 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:21.222071 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:22.941062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:22.941009 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:23.342535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:23.339566 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:23.561487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:23.561436 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:24.696775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:24.696744 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:24.907591 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:24.907557 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:26.415051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:26.415011 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:28.237636 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:28.237604 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:29.995693 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:29.995657 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:31.712964 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:31.712931 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:33.322104 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:33.322062 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:35.078441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.078244 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:35.314012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.313965 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:35.550444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.549567 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:35.666823 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.666794 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6"] Oct 30 04:50:35.671674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.671650 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:35.693212 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.693187 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6"] Oct 30 04:50:35.701179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.701135 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"openshift-service-ca.crt\"" Oct 30 04:50:35.701179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.701135 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"kube-root-ca.crt\"" Oct 30 04:50:35.701179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.701150 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"payload-processing-external-model-plugins-e2e-extmod-b-503804\"" Oct 30 04:50:35.701402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.701224 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"routing-overlay\"" Oct 30 04:50:35.701402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.701223 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"agc-cred-259031349671\"" Oct 30 04:50:35.701402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.701362 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-503804\"/\"payload-processing-external-model-e2e--0b0ba164-dockercfg-xpm8t\"" Oct 30 04:50:35.751515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.751474 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:35.751515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.751514 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg98k\" (UniqueName: \"kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-kube-api-access-qg98k\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:35.751775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.751698 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:35.751840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.751811 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:35.853407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.853310 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:35.854396 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.854287 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:35.854704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.854676 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:35.854855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.854721 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg98k\" (UniqueName: \"kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-kube-api-access-qg98k\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:35.854855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.854851 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:35.855658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.855635 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:35.878301 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:35.878269 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:36.026822 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:36.026772 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg98k\" (UniqueName: \"kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-kube-api-access-qg98k\") pod \"payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:36.286729 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:36.286698 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:36.595021 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:50:36.594990 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e20f87c_9cb2_44ab_8672_6f00395b1712.slice/crio-22a3a4970f2c8789db074f7b9ee84d8ba87a142771e155dd35f5c2c0b350f891 WatchSource:0}: Error finding container 22a3a4970f2c8789db074f7b9ee84d8ba87a142771e155dd35f5c2c0b350f891: Status 404 returned error can't find the container with id 22a3a4970f2c8789db074f7b9ee84d8ba87a142771e155dd35f5c2c0b350f891 Oct 30 04:50:36.595961 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:36.595938 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6"] Oct 30 04:50:37.022779 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:37.022738 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:37.173863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:37.173824 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" event={"ID":"7e20f87c-9cb2-44ab-8672-6f00395b1712","Type":"ContainerStarted","Data":"0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e"} Oct 30 04:50:37.174078 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:37.173887 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" event={"ID":"7e20f87c-9cb2-44ab-8672-6f00395b1712","Type":"ContainerStarted","Data":"22a3a4970f2c8789db074f7b9ee84d8ba87a142771e155dd35f5c2c0b350f891"} Oct 30 04:50:37.174078 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:37.173959 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:37.217153 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:37.217104 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" podStartSLOduration=2.217090148 podStartE2EDuration="2.217090148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:50:37.215129422 +0000 UTC m=+3160.501004197" watchObservedRunningTime="2026-10-30 04:50:37.217090148 +0000 UTC m=+3160.502964992" Oct 30 04:50:38.682683 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:38.682637 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:40.472354 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:40.472312 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:41.220459 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:50:41.220413 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:50:42.162412 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:42.162376 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:42.374397 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:42.374360 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:44.020057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:44.020019 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:45.642917 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:45.642879 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:47.253849 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:47.253814 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6"] Oct 30 04:50:47.256673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:47.256647 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:47.347813 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:47.347777 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:47.578361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:47.578279 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:47.818442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:47.810237 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:48.039600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:48.039564 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:49.628894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:49.628853 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:51.320996 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:51.320964 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:52.254498 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.254456 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" podUID="7e20f87c-9cb2-44ab-8672-6f00395b1712" containerName="payload-processing" containerID="cri-o://0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e" gracePeriod=25 Oct 30 04:50:52.528093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.528059 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:52.613913 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.613876 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-plugins-config-volume\") pod \"7e20f87c-9cb2-44ab-8672-6f00395b1712\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " Oct 30 04:50:52.614097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.613975 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-kube-api-access-qg98k\" (UniqueName: \"kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-kube-api-access-qg98k\") pod \"7e20f87c-9cb2-44ab-8672-6f00395b1712\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " Oct 30 04:50:52.614097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.614011 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-routing-overlay-volume\") pod \"7e20f87c-9cb2-44ab-8672-6f00395b1712\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " Oct 30 04:50:52.614323 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.614297 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-provider-credentials\") pod \"7e20f87c-9cb2-44ab-8672-6f00395b1712\" (UID: \"7e20f87c-9cb2-44ab-8672-6f00395b1712\") " Oct 30 04:50:52.614455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.614385 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-plugins-config-volume" pod "7e20f87c-9cb2-44ab-8672-6f00395b1712" (UID: "7e20f87c-9cb2-44ab-8672-6f00395b1712"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:50:52.614541 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.614515 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-routing-overlay-volume" pod "7e20f87c-9cb2-44ab-8672-6f00395b1712" (UID: "7e20f87c-9cb2-44ab-8672-6f00395b1712"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:50:52.615005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.614977 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:50:52.615115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.615009 4793 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7e20f87c-9cb2-44ab-8672-6f00395b1712-routing-overlay-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:50:52.616860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.616828 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-kube-api-access-qg98k" pod "7e20f87c-9cb2-44ab-8672-6f00395b1712" (UID: "7e20f87c-9cb2-44ab-8672-6f00395b1712"). InnerVolumeSpecName "kube-api-access-qg98k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:50:52.616951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.616865 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-provider-credentials" pod "7e20f87c-9cb2-44ab-8672-6f00395b1712" (UID: "7e20f87c-9cb2-44ab-8672-6f00395b1712"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:50:52.716497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.716472 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg98k\" (UniqueName: \"kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-kube-api-access-qg98k\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:50:52.716497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:52.716499 4793 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7e20f87c-9cb2-44ab-8672-6f00395b1712-provider-credentials\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:50:53.005742 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:53.005707 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:53.303392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:53.303355 4793 generic.go:356] "Generic (PLEG): container finished" podID="7e20f87c-9cb2-44ab-8672-6f00395b1712" containerID="0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e" exitCode=0 Oct 30 04:50:53.303590 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:53.303411 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" Oct 30 04:50:53.303590 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:53.303453 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" event={"ID":"7e20f87c-9cb2-44ab-8672-6f00395b1712","Type":"ContainerDied","Data":"0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e"} Oct 30 04:50:53.303590 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:53.303521 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6" event={"ID":"7e20f87c-9cb2-44ab-8672-6f00395b1712","Type":"ContainerDied","Data":"22a3a4970f2c8789db074f7b9ee84d8ba87a142771e155dd35f5c2c0b350f891"} Oct 30 04:50:53.303590 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:53.303549 4793 scope.go:122] "RemoveContainer" containerID="0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e" Oct 30 04:50:53.318987 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:53.317173 4793 scope.go:122] "RemoveContainer" containerID="0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e" Oct 30 04:50:53.319598 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:50:53.319569 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e\": container with ID starting with 0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e not found: ID does not exist" containerID="0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e" Oct 30 04:50:53.319782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:53.319745 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e"} err="failed to get container status \"0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e\": rpc error: code = NotFound desc = could not find container \"0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e\": container with ID starting with 0e005c69101b3968de7f8f20842f313ad9faa555176a5d43155d5e325b4e0d8e not found: ID does not exist" Oct 30 04:50:53.361187 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:53.361160 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6"] Oct 30 04:50:53.374012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:53.373985 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-503804/payload-processing-external-model-e2e-extmod-b-503804-6f95hbqz6"] Oct 30 04:50:54.113451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:54.111921 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:54.323580 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:54.323542 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:54.724114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:54.724074 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:54.847232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:54.847194 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e20f87c-9cb2-44ab-8672-6f00395b1712" path="/var/lib/kubelet/pods/7e20f87c-9cb2-44ab-8672-6f00395b1712/volumes" Oct 30 04:50:56.405724 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:56.405687 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:56.628808 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:56.628773 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:58.078725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:58.078687 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:50:59.698609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:50:59.698570 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:00.381536 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:00.381499 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:00.603750 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:00.603715 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:00.839489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:00.839407 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:01.085640 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:01.085596 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:01.307763 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:01.307728 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:01.521549 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:01.521511 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:01.882622 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:01.882587 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:02.209539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:02.209505 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:03.320914 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.320868 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:03.546407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.546378 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t"] Oct 30 04:51:03.547103 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.547087 4793 cpu_manager.go:414] "containerMap: removing container" podUID="7e20f87c-9cb2-44ab-8672-6f00395b1712" containerName="payload-processing" Oct 30 04:51:03.547150 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.547105 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e20f87c-9cb2-44ab-8672-6f00395b1712" containerName="payload-processing" Oct 30 04:51:03.547204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.547195 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e20f87c-9cb2-44ab-8672-6f00395b1712" containerName="payload-processing" Oct 30 04:51:03.550687 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.550662 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.557918 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.557894 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:03.567953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.567932 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-6378cc\"" Oct 30 04:51:03.572546 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.572489 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t"] Oct 30 04:51:03.578935 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.578912 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"agc-cred-b19d83c3194d\"" Oct 30 04:51:03.579492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.579476 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"payload-processing-external-model-e2e--18641857-dockercfg-9hk4x\"" Oct 30 04:51:03.681755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.681718 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"openshift-service-ca.crt\"" Oct 30 04:51:03.682268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.682072 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"routing-overlay\"" Oct 30 04:51:03.682402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.682177 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-6378cc\"/\"kube-root-ca.crt\"" Oct 30 04:51:03.702519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.702470 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.702955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.702927 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wrtd\" (UniqueName: \"kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-kube-api-access-5wrtd\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.703129 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.703065 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.703199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.703150 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.804611 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.804578 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.804814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.804665 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.804814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.804748 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.804950 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.804848 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wrtd\" (UniqueName: \"kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-kube-api-access-5wrtd\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.805324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.805294 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.805488 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.805439 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.807194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.807170 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.828832 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.828780 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wrtd\" (UniqueName: \"kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-kube-api-access-5wrtd\") pod \"payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:03.866846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:03.866825 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:04.172746 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:51:04.172711 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f88f8f_4fd7_4bbb_bcd9_0104e14dcea4.slice/crio-f9a29092dd1a75d8dff113a40b8f9fccf400db80faff88b6be518c363eaf7437 WatchSource:0}: Error finding container f9a29092dd1a75d8dff113a40b8f9fccf400db80faff88b6be518c363eaf7437: Status 404 returned error can't find the container with id f9a29092dd1a75d8dff113a40b8f9fccf400db80faff88b6be518c363eaf7437 Oct 30 04:51:04.173283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:04.173214 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t"] Oct 30 04:51:04.423801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:04.423722 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" event={"ID":"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4","Type":"ContainerStarted","Data":"a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696"} Oct 30 04:51:04.423801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:04.423784 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:04.423801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:04.423801 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" event={"ID":"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4","Type":"ContainerStarted","Data":"f9a29092dd1a75d8dff113a40b8f9fccf400db80faff88b6be518c363eaf7437"} Oct 30 04:51:04.435633 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:04.435595 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:04.497664 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:04.486752 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" podStartSLOduration=1.486727883 podStartE2EDuration="1.486727883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:04.467548867 +0000 UTC m=+3187.753423664" watchObservedRunningTime="2026-10-30 04:51:04.486727883 +0000 UTC m=+3187.772602675" Oct 30 04:51:04.682501 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:04.680756 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:05.074143 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:05.070250 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:05.289369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:05.289305 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:05.546456 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:05.545693 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6378cc/agc-cred-b19d83c3194d: secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:05.546456 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:05.545722 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t: secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:05.546456 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:05.545780 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials podName:64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4 nodeName:}" failed. No retries permitted until 2026-10-30 04:51:06.045767077 +0000 UTC m=+3189.331641834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" (UID: "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4") : secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:06.053079 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:06.053052 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6378cc/agc-cred-b19d83c3194d: secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:06.053079 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:06.053075 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t: secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:06.053288 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:06.053137 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials podName:64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4 nodeName:}" failed. No retries permitted until 2026-10-30 04:51:07.053121838 +0000 UTC m=+3190.338996591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" (UID: "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4") : secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:07.081614 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:07.081573 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6378cc/agc-cred-b19d83c3194d: secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:07.081614 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:07.081606 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t: secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:07.082056 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:07.081667 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials podName:64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4 nodeName:}" failed. No retries permitted until 2026-10-30 04:51:09.081648096 +0000 UTC m=+3192.367522853 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" (UID: "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4") : secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:07.087868 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:07.087816 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:08.999253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:08.999199 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:09.108799 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:09.108771 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6378cc/agc-cred-b19d83c3194d: secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:09.108799 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:09.108795 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t: secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:09.109007 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:09.108847 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials podName:64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4 nodeName:}" failed. No retries permitted until 2026-10-30 04:51:13.108832416 +0000 UTC m=+3196.394707169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" (UID: "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4") : secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:09.374649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:09.374543 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:10.769215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:10.769077 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:11.048807 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.048727 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:11.123972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.123937 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf"] Oct 30 04:51:11.129916 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.129890 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.147712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.147690 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fcf42a-openshift-default-dockercfg-rv9xg\"" Oct 30 04:51:11.164893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.164867 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf"] Oct 30 04:51:11.226606 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:11.226568 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:51:11.237234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.236050 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba822078-a0dd-462f-bd7d-add7a00496b1-istiod-ca-cert\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.237234 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.236144 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-socket\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.237947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.237536 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-token\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.237947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.237701 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-data\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.237947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.237880 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-envoy\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.241851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.241224 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-podinfo\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.241851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.241275 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-credential-socket\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.241851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.241489 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg7cn\" (UniqueName: \"kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-kube-api-access-pg7cn\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.241851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.241580 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-certs\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.331521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.331368 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:11.348073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.348035 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg7cn\" (UniqueName: \"kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-kube-api-access-pg7cn\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.348355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.348138 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-certs\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.348355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.348197 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba822078-a0dd-462f-bd7d-add7a00496b1-istiod-ca-cert\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.348355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.348312 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-socket\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.348594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.348386 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-token\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.348594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.348438 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-data\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.348594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.348512 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-envoy\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.348594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.348565 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-podinfo\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.348833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.348593 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-credential-socket\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.349281 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.349249 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-credential-socket\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.356975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.354410 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-certs\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.356975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.356345 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba822078-a0dd-462f-bd7d-add7a00496b1-istiod-ca-cert\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.356975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.356743 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-socket\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.360350 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.358813 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-data\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.364166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.364139 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-envoy\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.387063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.387009 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-token\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.392817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.391976 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-podinfo\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.403601 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.403546 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg7cn\" (UniqueName: \"kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-kube-api-access-pg7cn\") pod \"e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.444871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.444838 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:11.648575 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.648485 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:11.767036 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:51:11.764734 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba822078_a0dd_462f_bd7d_add7a00496b1.slice/crio-5e1264f9d51c2eb3679c732639495452bea6eefba5a5e2b1ba9a2e41d9d4dcae WatchSource:0}: Error finding container 5e1264f9d51c2eb3679c732639495452bea6eefba5a5e2b1ba9a2e41d9d4dcae: Status 404 returned error can't find the container with id 5e1264f9d51c2eb3679c732639495452bea6eefba5a5e2b1ba9a2e41d9d4dcae Oct 30 04:51:11.767036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.766060 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf"] Oct 30 04:51:11.770626 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.770588 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:51:11.771098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.770664 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:51:11.771098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:11.770692 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:51:12.356585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:12.356552 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:12.484662 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:12.484624 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" event={"ID":"ba822078-a0dd-462f-bd7d-add7a00496b1","Type":"ContainerStarted","Data":"59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921"} Oct 30 04:51:12.484846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:12.484714 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" event={"ID":"ba822078-a0dd-462f-bd7d-add7a00496b1","Type":"ContainerStarted","Data":"5e1264f9d51c2eb3679c732639495452bea6eefba5a5e2b1ba9a2e41d9d4dcae"} Oct 30 04:51:12.554258 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:12.554201 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" podStartSLOduration=1.554184008 podStartE2EDuration="1.554184008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:12.546928067 +0000 UTC m=+3195.832802841" watchObservedRunningTime="2026-10-30 04:51:12.554184008 +0000 UTC m=+3195.840058785" Oct 30 04:51:12.573053 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:12.573006 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:12.806622 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:12.806585 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:13.180206 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:13.180176 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6378cc/agc-cred-b19d83c3194d: secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:13.180206 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:13.180199 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t: secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:13.180412 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:13.180259 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials podName:64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4 nodeName:}" failed. No retries permitted until 2026-10-30 04:51:21.180244823 +0000 UTC m=+3204.466119592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" (UID: "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4") : secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:13.445175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:13.445091 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:13.450664 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:13.450635 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:13.491380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:13.491351 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:13.492092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:13.492072 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:51:14.727711 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:14.727672 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:15.221036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:15.221005 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:15.434908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:15.434883 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:15.516316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:15.516229 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:16.888194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:16.888163 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:17.190839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.190804 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:17.513017 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.512932 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t"] Oct 30 04:51:17.826490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.825046 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:17.836504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.836468 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg"] Oct 30 04:51:17.869247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.869216 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:17.869592 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.869365 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg"] Oct 30 04:51:17.885517 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.885484 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c0e964-openshift-default-dockercfg-k2x68\"" Oct 30 04:51:17.956077 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.956050 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmmc\" (UniqueName: \"kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-kube-api-access-5lmmc\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:17.956539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.956091 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990dd3c0-bdfb-4779-bf61-df951f964d9f-istiod-ca-cert\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:17.956539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.956272 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-socket\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:17.956539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.956372 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-credential-socket\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:17.956539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.956497 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-certs\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:17.956769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.956548 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-token\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:17.956769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.956721 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-podinfo\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:17.956889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.956799 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-data\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:17.956889 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:17.956847 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-envoy\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.058563 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.058534 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-envoy\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.058753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.058590 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmmc\" (UniqueName: \"kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-kube-api-access-5lmmc\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.058753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.058627 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990dd3c0-bdfb-4779-bf61-df951f964d9f-istiod-ca-cert\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.058753 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.058715 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-socket\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.058949 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.058763 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-credential-socket\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.058949 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.058841 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-certs\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.058949 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.058889 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-token\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.059119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.058980 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-podinfo\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.059119 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.059032 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-data\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.059319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.059288 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-socket\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.059319 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.059308 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-certs\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.059565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.059535 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-credential-socket\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.059713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.059662 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990dd3c0-bdfb-4779-bf61-df951f964d9f-istiod-ca-cert\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.059786 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.059720 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-data\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.060987 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.060965 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-envoy\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.061568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.061545 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-podinfo\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.072902 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.072874 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:18.086479 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.086410 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmmc\" (UniqueName: \"kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-kube-api-access-5lmmc\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.087089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.087068 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-token\") pod \"e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.187456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.187393 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:18.446640 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.446607 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd"] Oct 30 04:51:18.511082 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.511035 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:18.532891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.532865 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg"] Oct 30 04:51:18.533552 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:51:18.533527 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990dd3c0_bdfb_4779_bf61_df951f964d9f.slice/crio-63f81600f48b4e6e9306976601aeed9b74ea1f339dd68467a876948a67d3fcd2 WatchSource:0}: Error finding container 63f81600f48b4e6e9306976601aeed9b74ea1f339dd68467a876948a67d3fcd2: Status 404 returned error can't find the container with id 63f81600f48b4e6e9306976601aeed9b74ea1f339dd68467a876948a67d3fcd2 Oct 30 04:51:18.536614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.536580 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:51:18.536688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.536645 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:51:18.536688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.536674 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:51:18.747298 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:18.747207 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:19.002494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:19.002280 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:19.543371 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:19.543333 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" event={"ID":"990dd3c0-bdfb-4779-bf61-df951f964d9f","Type":"ContainerStarted","Data":"e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f"} Oct 30 04:51:19.543581 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:19.543434 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" event={"ID":"990dd3c0-bdfb-4779-bf61-df951f964d9f","Type":"ContainerStarted","Data":"63f81600f48b4e6e9306976601aeed9b74ea1f339dd68467a876948a67d3fcd2"} Oct 30 04:51:19.617616 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:19.617558 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" podStartSLOduration=2.617540963 podStartE2EDuration="2.617540963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:19.616801606 +0000 UTC m=+3202.902676380" watchObservedRunningTime="2026-10-30 04:51:19.617540963 +0000 UTC m=+3202.903415739" Oct 30 04:51:20.188480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:20.188433 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:20.190490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:20.190454 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Oct 30 04:51:20.190579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:20.190517 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" podUID="990dd3c0-bdfb-4779-bf61-df951f964d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Oct 30 04:51:20.624510 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:20.623862 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:21.188975 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:21.188944 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Oct 30 04:51:21.189362 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:21.188999 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" podUID="990dd3c0-bdfb-4779-bf61-df951f964d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Oct 30 04:51:21.222188 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:21.222160 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-6378cc/agc-cred-b19d83c3194d: secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:21.222188 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:21.222183 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t: secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:21.222368 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:21.222232 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials podName:64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4 nodeName:}" failed. No retries permitted until 2026-10-30 04:51:37.222217996 +0000 UTC m=+3220.508092769 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" (UID: "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4") : secret "agc-cred-b19d83c3194d" not found Oct 30 04:51:22.188310 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.188271 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Oct 30 04:51:22.188566 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.188335 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" podUID="990dd3c0-bdfb-4779-bf61-df951f964d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Oct 30 04:51:22.334155 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.334122 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:22.514003 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.513914 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" podUID="64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4" containerName="payload-processing" containerID="cri-o://a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696" gracePeriod=25 Oct 30 04:51:22.677440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.673807 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:22.846944 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.846920 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:22.944356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.944324 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-kube-api-access-5wrtd\" (UniqueName: \"kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-kube-api-access-5wrtd\") pod \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " Oct 30 04:51:22.944556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.944456 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials\") pod \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " Oct 30 04:51:22.944556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.944480 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-plugins-config-volume\") pod \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " Oct 30 04:51:22.944556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.944555 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-routing-overlay-volume\") pod \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\" (UID: \"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4\") " Oct 30 04:51:22.945026 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.944998 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-plugins-config-volume" pod "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4" (UID: "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:51:22.945377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.945359 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:51:22.945831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.945810 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-routing-overlay-volume" pod "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4" (UID: "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:51:22.947302 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.947274 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-kube-api-access-5wrtd" pod "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4" (UID: "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4"). InnerVolumeSpecName "kube-api-access-5wrtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:51:22.947628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:22.947602 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials" pod "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4" (UID: "64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:51:23.046866 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.046836 4793 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-provider-credentials\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:51:23.046866 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.046862 4793 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-routing-overlay-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:51:23.046866 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.046871 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wrtd\" (UniqueName: \"kubernetes.io/projected/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4-kube-api-access-5wrtd\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:51:23.187886 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.187852 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Oct 30 04:51:23.188089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.187915 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" podUID="990dd3c0-bdfb-4779-bf61-df951f964d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Oct 30 04:51:23.447666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.447559 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" podUID="7b0e8b2e-bf65-4209-9358-7f69e37e4e28" containerName="payload-processing" containerID="cri-o://92d6b41af503b89b04fed8af7fca50661e3cba2d395dd98b63919abd9217b209" gracePeriod=25 Oct 30 04:51:23.603467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.603245 4793 generic.go:356] "Generic (PLEG): container finished" podID="64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4" containerID="a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696" exitCode=0 Oct 30 04:51:23.603467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.603319 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" Oct 30 04:51:23.603467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.603323 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" event={"ID":"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4","Type":"ContainerDied","Data":"a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696"} Oct 30 04:51:23.603467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.603457 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t" event={"ID":"64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4","Type":"ContainerDied","Data":"f9a29092dd1a75d8dff113a40b8f9fccf400db80faff88b6be518c363eaf7437"} Oct 30 04:51:23.603782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.603492 4793 scope.go:122] "RemoveContainer" containerID="a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696" Oct 30 04:51:23.606251 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.606224 4793 generic.go:356] "Generic (PLEG): container finished" podID="7b0e8b2e-bf65-4209-9358-7f69e37e4e28" containerID="92d6b41af503b89b04fed8af7fca50661e3cba2d395dd98b63919abd9217b209" exitCode=0 Oct 30 04:51:23.606578 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.606556 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" event={"ID":"7b0e8b2e-bf65-4209-9358-7f69e37e4e28","Type":"ContainerDied","Data":"92d6b41af503b89b04fed8af7fca50661e3cba2d395dd98b63919abd9217b209"} Oct 30 04:51:23.616838 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.616820 4793 scope.go:122] "RemoveContainer" containerID="a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696" Oct 30 04:51:23.617089 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:23.617068 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696\": container with ID starting with a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696 not found: ID does not exist" containerID="a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696" Oct 30 04:51:23.617160 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.617098 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696"} err="failed to get container status \"a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696\": rpc error: code = NotFound desc = could not find container \"a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696\": container with ID starting with a948415a89565400beb6f109b75fbea50e5e02076888f950afc4ff49e96d1696 not found: ID does not exist" Oct 30 04:51:23.701952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.701859 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t"] Oct 30 04:51:23.707543 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.707512 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-6378cc/payload-processing-external-model-e2e-extmod-a-6378cc-6c95vf97t"] Oct 30 04:51:23.847617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:23.847590 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:51:24.010030 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.009948 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-plugins-config-volume\") pod \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " Oct 30 04:51:24.010795 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.010662 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-plugins-config-volume" pod "7b0e8b2e-bf65-4209-9358-7f69e37e4e28" (UID: "7b0e8b2e-bf65-4209-9358-7f69e37e4e28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:51:24.011017 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.010995 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-kube-api-access-m4pr9\" (UniqueName: \"kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-kube-api-access-m4pr9\") pod \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " Oct 30 04:51:24.011317 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.011299 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-routing-overlay-volume\") pod \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " Oct 30 04:51:24.011518 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.011504 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-provider-credentials\") pod \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\" (UID: \"7b0e8b2e-bf65-4209-9358-7f69e37e4e28\") " Oct 30 04:51:24.011828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.011703 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-routing-overlay-volume" pod "7b0e8b2e-bf65-4209-9358-7f69e37e4e28" (UID: "7b0e8b2e-bf65-4209-9358-7f69e37e4e28"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:51:24.012457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.012436 4793 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-routing-overlay-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:51:24.012581 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.012569 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:51:24.014539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.014473 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-kube-api-access-m4pr9" pod "7b0e8b2e-bf65-4209-9358-7f69e37e4e28" (UID: "7b0e8b2e-bf65-4209-9358-7f69e37e4e28"). InnerVolumeSpecName "kube-api-access-m4pr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:51:24.015048 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.014930 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:24.015577 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.015540 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-provider-credentials" pod "7b0e8b2e-bf65-4209-9358-7f69e37e4e28" (UID: "7b0e8b2e-bf65-4209-9358-7f69e37e4e28"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:51:24.114450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.114395 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4pr9\" (UniqueName: \"kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-kube-api-access-m4pr9\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:51:24.114450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.114439 4793 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7b0e8b2e-bf65-4209-9358-7f69e37e4e28-provider-credentials\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:51:24.188266 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.188229 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" start-of-body= Oct 30 04:51:24.188497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.188295 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" podUID="990dd3c0-bdfb-4779-bf61-df951f964d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.214:15021/healthz/ready\": dial tcp 10.133.0.214:15021: connect: connection refused" Oct 30 04:51:24.508449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.508111 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:24.615003 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.614972 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" Oct 30 04:51:24.615184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.615000 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd" event={"ID":"7b0e8b2e-bf65-4209-9358-7f69e37e4e28","Type":"ContainerDied","Data":"3994dbfee68a5f7d16c1b29d8ebbc6ed2b342e67748f1f5b2d59c752c794d7f5"} Oct 30 04:51:24.615184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.615061 4793 scope.go:122] "RemoveContainer" containerID="92d6b41af503b89b04fed8af7fca50661e3cba2d395dd98b63919abd9217b209" Oct 30 04:51:24.676220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.676189 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd"] Oct 30 04:51:24.688671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.688646 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-05bee8/payload-processing-external-model-e2e-extmod-a-05bee8-b959h26nd"] Oct 30 04:51:24.738574 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.738534 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:24.847481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.847393 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4" path="/var/lib/kubelet/pods/64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4/volumes" Oct 30 04:51:24.847948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:24.847930 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0e8b2e-bf65-4209-9358-7f69e37e4e28" path="/var/lib/kubelet/pods/7b0e8b2e-bf65-4209-9358-7f69e37e4e28/volumes" Oct 30 04:51:25.192963 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:25.192931 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:25.193316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:25.193286 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:25.194061 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:25.194042 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 04:51:25.725506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:25.725472 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:27.520435 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:27.520387 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:29.081732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:29.081690 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:30.849952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:30.849913 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:31.328464 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:31.328388 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:31.558434 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:31.558392 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:32.135245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:32.135207 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:32.357440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:32.356839 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:32.891076 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:32.890974 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:33.040563 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.040528 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk"] Oct 30 04:51:33.041221 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.041206 4793 cpu_manager.go:414] "containerMap: removing container" podUID="7b0e8b2e-bf65-4209-9358-7f69e37e4e28" containerName="payload-processing" Oct 30 04:51:33.041270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.041223 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0e8b2e-bf65-4209-9358-7f69e37e4e28" containerName="payload-processing" Oct 30 04:51:33.041270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.041253 4793 cpu_manager.go:414] "containerMap: removing container" podUID="64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4" containerName="payload-processing" Oct 30 04:51:33.041270 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.041259 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4" containerName="payload-processing" Oct 30 04:51:33.041370 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.041360 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0e8b2e-bf65-4209-9358-7f69e37e4e28" containerName="payload-processing" Oct 30 04:51:33.041404 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.041373 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f88f8f-4fd7-4bbb-bcd9-0104e14dcea4" containerName="payload-processing" Oct 30 04:51:33.045063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.045045 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.065577 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.065541 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk"] Oct 30 04:51:33.083456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.072982 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 04:51:33.083456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.073061 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 04:51:33.083456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.073207 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 04:51:33.083456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.073257 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9b41a77802f1\"" Oct 30 04:51:33.083456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.073618 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 04:51:33.083456 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.074572 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-42vq2\"" Oct 30 04:51:33.105461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.105430 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:33.173548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.168479 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-plugins-config-volume\") pod \"payload-processing-external-model-56fd9d7c8c-bjhwk\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.173548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.168593 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-routing-overlay-volume\") pod \"payload-processing-external-model-56fd9d7c8c-bjhwk\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.173548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.168671 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns8tv\" (UniqueName: \"kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-kube-api-access-ns8tv\") pod \"payload-processing-external-model-56fd9d7c8c-bjhwk\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.173548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.168771 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-provider-credentials\") pod \"payload-processing-external-model-56fd9d7c8c-bjhwk\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.303446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.286079 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-plugins-config-volume\") pod \"payload-processing-external-model-56fd9d7c8c-bjhwk\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.303446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.286196 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-routing-overlay-volume\") pod \"payload-processing-external-model-56fd9d7c8c-bjhwk\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.303446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.286278 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns8tv\" (UniqueName: \"kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-kube-api-access-ns8tv\") pod \"payload-processing-external-model-56fd9d7c8c-bjhwk\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.303446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.286323 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-provider-credentials\") pod \"payload-processing-external-model-56fd9d7c8c-bjhwk\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.303446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.287606 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-plugins-config-volume\") pod \"payload-processing-external-model-56fd9d7c8c-bjhwk\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.303446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.288677 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-routing-overlay-volume\") pod \"payload-processing-external-model-56fd9d7c8c-bjhwk\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.326439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.315623 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns8tv\" (UniqueName: \"kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-kube-api-access-ns8tv\") pod \"payload-processing-external-model-56fd9d7c8c-bjhwk\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.333437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.328476 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-provider-credentials\") pod \"payload-processing-external-model-56fd9d7c8c-bjhwk\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.361296 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.361035 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:33.677469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.677375 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk"] Oct 30 04:51:33.678036 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:51:33.678011 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d6ee99_51cf_4321_946b_df3e96e80ac6.slice/crio-3ca161cb75b5d5008bf3cfe7c6bb0f19a242e26076d6598f7e08a0b5ac59be40 WatchSource:0}: Error finding container 3ca161cb75b5d5008bf3cfe7c6bb0f19a242e26076d6598f7e08a0b5ac59be40: Status 404 returned error can't find the container with id 3ca161cb75b5d5008bf3cfe7c6bb0f19a242e26076d6598f7e08a0b5ac59be40 Oct 30 04:51:33.689073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:33.689043 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" event={"ID":"b2d6ee99-51cf-4321-946b-df3e96e80ac6","Type":"ContainerStarted","Data":"3ca161cb75b5d5008bf3cfe7c6bb0f19a242e26076d6598f7e08a0b5ac59be40"} Oct 30 04:51:34.699754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:34.699710 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" event={"ID":"b2d6ee99-51cf-4321-946b-df3e96e80ac6","Type":"ContainerStarted","Data":"5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1"} Oct 30 04:51:34.700247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:34.699805 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:34.751529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:34.751451 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" podStartSLOduration=1.751405533 podStartE2EDuration="1.751405533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:51:34.747376864 +0000 UTC m=+3218.033251639" watchObservedRunningTime="2026-10-30 04:51:34.751405533 +0000 UTC m=+3218.037280376" Oct 30 04:51:34.821086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:34.821040 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:36.500560 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:36.500519 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:38.223708 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:38.223674 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:39.910030 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:39.909994 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:41.220262 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:51:41.220229 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:51:41.677864 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:41.677828 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:43.473109 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:43.473069 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:45.110618 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:45.110583 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:45.330890 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:45.330854 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:45.711280 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:45.711252 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:51:46.703211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:46.703182 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:47.410565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:47.410524 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:47.623851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:47.623812 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:48.408809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:48.408769 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:50.008332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:50.008291 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:51.654817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:51.654772 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:53.363824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:53.363786 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:55.121069 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:55.121031 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:55.339870 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:55.339821 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:55.572432 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:55.572392 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:56.648113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:56.648076 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:56.864445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:56.864187 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-78prl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:51:56.864445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:56.864270 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" podUID="a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:51:56.869637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:56.869522 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:57.173064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:57.173016 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:57.403947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:57.403860 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:57.629731 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:57.629693 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:57.845801 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:57.845761 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:58.076815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:58.076777 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:58.531136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:58.531098 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:51:59.109369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:51:59.109278 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:00.771355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:00.771295 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:02.437860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:02.437820 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:04.077642 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:04.077461 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:04.313116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:04.313069 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:04.654749 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:04.654713 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:04.864366 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:04.864326 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:05.909180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:05.909146 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:07.656594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:07.656557 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:09.347239 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:09.347197 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:09.476201 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:09.476168 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk"] Oct 30 04:52:09.610678 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:09.610591 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:09.852344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:09.852303 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:10.873395 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:10.873363 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:11.137463 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:11.137350 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:11.222018 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:52:11.221985 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:52:12.446327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:12.446287 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:12.690611 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:12.690577 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:12.891505 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:12.891386 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:13.134927 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:13.134878 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:13.348647 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:13.348609 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:13.618219 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:13.618137 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:13.817833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:13.817789 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:14.476720 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.476680 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" podUID="b2d6ee99-51cf-4321-946b-df3e96e80ac6" containerName="payload-processing" containerID="cri-o://5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1" gracePeriod=25 Oct 30 04:52:14.757121 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.757095 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:52:14.897070 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.897040 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-kube-api-access-ns8tv\" (UniqueName: \"kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-kube-api-access-ns8tv\") pod \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " Oct 30 04:52:14.897268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.897093 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-routing-overlay-volume\") pod \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " Oct 30 04:52:14.897268 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.897179 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-plugins-config-volume\") pod \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " Oct 30 04:52:14.897394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.897333 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-provider-credentials\") pod \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\" (UID: \"b2d6ee99-51cf-4321-946b-df3e96e80ac6\") " Oct 30 04:52:14.897532 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.897502 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-routing-overlay-volume" pod "b2d6ee99-51cf-4321-946b-df3e96e80ac6" (UID: "b2d6ee99-51cf-4321-946b-df3e96e80ac6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:52:14.897840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.897813 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-plugins-config-volume" pod "b2d6ee99-51cf-4321-946b-df3e96e80ac6" (UID: "b2d6ee99-51cf-4321-946b-df3e96e80ac6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:52:14.897970 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.897952 4793 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-routing-overlay-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:52:14.898029 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.897978 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d6ee99-51cf-4321-946b-df3e96e80ac6-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:52:14.899635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.899614 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-provider-credentials" pod "b2d6ee99-51cf-4321-946b-df3e96e80ac6" (UID: "b2d6ee99-51cf-4321-946b-df3e96e80ac6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:52:14.899734 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.899717 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-kube-api-access-ns8tv" pod "b2d6ee99-51cf-4321-946b-df3e96e80ac6" (UID: "b2d6ee99-51cf-4321-946b-df3e96e80ac6"). InnerVolumeSpecName "kube-api-access-ns8tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:52:14.999079 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.998976 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns8tv\" (UniqueName: \"kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-kube-api-access-ns8tv\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:52:14.999079 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:14.999031 4793 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2d6ee99-51cf-4321-946b-df3e96e80ac6-provider-credentials\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:52:15.075353 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.075311 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:15.077012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.076979 4793 generic.go:356] "Generic (PLEG): container finished" podID="b2d6ee99-51cf-4321-946b-df3e96e80ac6" containerID="5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1" exitCode=0 Oct 30 04:52:15.077173 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.077148 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" event={"ID":"b2d6ee99-51cf-4321-946b-df3e96e80ac6","Type":"ContainerDied","Data":"5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1"} Oct 30 04:52:15.077248 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.077208 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" event={"ID":"b2d6ee99-51cf-4321-946b-df3e96e80ac6","Type":"ContainerDied","Data":"3ca161cb75b5d5008bf3cfe7c6bb0f19a242e26076d6598f7e08a0b5ac59be40"} Oct 30 04:52:15.077248 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.077240 4793 scope.go:122] "RemoveContainer" containerID="5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1" Oct 30 04:52:15.077478 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.077460 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk" Oct 30 04:52:15.091284 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.091266 4793 scope.go:122] "RemoveContainer" containerID="5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1" Oct 30 04:52:15.091644 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:52:15.091617 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1\": container with ID starting with 5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1 not found: ID does not exist" containerID="5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1" Oct 30 04:52:15.091737 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.091648 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1"} err="failed to get container status \"5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1\": rpc error: code = NotFound desc = could not find container \"5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1\": container with ID starting with 5c8f7f23ca688745b093aca97925d914760ca1d65b0ecffde9870e51675071a1 not found: ID does not exist" Oct 30 04:52:15.132722 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.132691 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk"] Oct 30 04:52:15.136619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.136595 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-56fd9d7c8c-bjhwk"] Oct 30 04:52:15.285114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.285028 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:15.356653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.356609 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs"] Oct 30 04:52:15.357330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.357312 4793 cpu_manager.go:414] "containerMap: removing container" podUID="b2d6ee99-51cf-4321-946b-df3e96e80ac6" containerName="payload-processing" Oct 30 04:52:15.357330 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.357330 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d6ee99-51cf-4321-946b-df3e96e80ac6" containerName="payload-processing" Oct 30 04:52:15.357484 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.357459 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d6ee99-51cf-4321-946b-df3e96e80ac6" containerName="payload-processing" Oct 30 04:52:15.362364 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.362343 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.382502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.381826 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"openshift-service-ca.crt\"" Oct 30 04:52:15.382502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.381852 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"agc-cred-3f798d461c2c\"" Oct 30 04:52:15.382502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.381902 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"routing-overlay\"" Oct 30 04:52:15.382502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.381946 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"payload-processing-external-model-plugins-e2e-extmod-b-483bea\"" Oct 30 04:52:15.382502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.382083 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"payload-processing-external-model-e2e--675c86f9-dockercfg-ftxjr\"" Oct 30 04:52:15.382502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.382285 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs"] Oct 30 04:52:15.383161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.383134 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-483bea\"/\"kube-root-ca.crt\"" Oct 30 04:52:15.506826 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.506795 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-8448985qs\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.506982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.506852 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-8448985qs\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.506982 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.506936 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-8448985qs\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.507061 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.506979 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqjjl\" (UniqueName: \"kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-kube-api-access-sqjjl\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-8448985qs\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.611522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.609976 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqjjl\" (UniqueName: \"kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-kube-api-access-sqjjl\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-8448985qs\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.625444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.622792 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-8448985qs\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.625444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.622902 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-8448985qs\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.625444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.622953 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-8448985qs\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.625810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.625630 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-8448985qs\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.626309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.626279 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-8448985qs\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.645276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.645248 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-8448985qs\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.677436 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.677244 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqjjl\" (UniqueName: \"kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-kube-api-access-sqjjl\") pod \"payload-processing-external-model-e2e-extmod-b-483bea-8448985qs\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:15.690860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:15.690815 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:16.032955 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:52:16.032926 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78dbe08d_e4b5_42b7_9438_055fd6504bff.slice/crio-4ff5eebfa30d66d1d24057907322faec9cdd5951b600545e3104053a2914be2e WatchSource:0}: Error finding container 4ff5eebfa30d66d1d24057907322faec9cdd5951b600545e3104053a2914be2e: Status 404 returned error can't find the container with id 4ff5eebfa30d66d1d24057907322faec9cdd5951b600545e3104053a2914be2e Oct 30 04:52:16.034288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:16.034261 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs"] Oct 30 04:52:16.088813 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:16.088787 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" event={"ID":"78dbe08d-e4b5-42b7-9438-055fd6504bff","Type":"ContainerStarted","Data":"4ff5eebfa30d66d1d24057907322faec9cdd5951b600545e3104053a2914be2e"} Oct 30 04:52:16.354515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:16.351746 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:16.857226 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:16.855984 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d6ee99-51cf-4321-946b-df3e96e80ac6" path="/var/lib/kubelet/pods/b2d6ee99-51cf-4321-946b-df3e96e80ac6/volumes" Oct 30 04:52:17.007810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:17.007772 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:17.099634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:17.099597 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" event={"ID":"78dbe08d-e4b5-42b7-9438-055fd6504bff","Type":"ContainerStarted","Data":"c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c"} Oct 30 04:52:17.100107 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:17.099810 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:17.148387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:17.148292 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" podStartSLOduration=2.148277142 podStartE2EDuration="2.148277142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:17.147281954 +0000 UTC m=+3260.433156730" watchObservedRunningTime="2026-10-30 04:52:17.148277142 +0000 UTC m=+3260.434151916" Oct 30 04:52:18.251102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:18.251070 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:18.476994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:18.476957 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:18.770686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:18.770638 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:19.289882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.289841 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:19.488637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.488601 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:19.887460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.887403 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf"] Oct 30 04:52:19.892172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.892150 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:19.918884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.918856 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3ad88dd1d8f2\"" Oct 30 04:52:19.919199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.919173 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-ddlgx\"" Oct 30 04:52:19.919413 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.919393 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 04:52:19.919685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.919667 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 04:52:19.919824 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.919183 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 04:52:19.920447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.920407 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 04:52:19.926929 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.926906 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf"] Oct 30 04:52:19.940192 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.940162 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdhxg\" (UniqueName: \"kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-kube-api-access-fdhxg\") pod \"payload-processing-external-model-6db5744bf9-qv2bf\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:19.940302 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.940247 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-provider-credentials\") pod \"payload-processing-external-model-6db5744bf9-qv2bf\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:19.940459 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.940401 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-routing-overlay-volume\") pod \"payload-processing-external-model-6db5744bf9-qv2bf\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:19.940609 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:19.940592 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-plugins-config-volume\") pod \"payload-processing-external-model-6db5744bf9-qv2bf\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:20.042295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:20.042256 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-provider-credentials\") pod \"payload-processing-external-model-6db5744bf9-qv2bf\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:20.042511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:20.042314 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-routing-overlay-volume\") pod \"payload-processing-external-model-6db5744bf9-qv2bf\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:20.042511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:20.042374 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-plugins-config-volume\") pod \"payload-processing-external-model-6db5744bf9-qv2bf\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:20.042511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:20.042490 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhxg\" (UniqueName: \"kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-kube-api-access-fdhxg\") pod \"payload-processing-external-model-6db5744bf9-qv2bf\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:20.043075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:20.043052 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-routing-overlay-volume\") pod \"payload-processing-external-model-6db5744bf9-qv2bf\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:20.043184 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:20.043171 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-plugins-config-volume\") pod \"payload-processing-external-model-6db5744bf9-qv2bf\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:20.044794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:20.044772 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-provider-credentials\") pod \"payload-processing-external-model-6db5744bf9-qv2bf\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:20.066342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:20.066313 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhxg\" (UniqueName: \"kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-kube-api-access-fdhxg\") pod \"payload-processing-external-model-6db5744bf9-qv2bf\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:20.225767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:20.225737 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:20.561384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:20.561348 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf"] Oct 30 04:52:20.563055 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:52:20.563023 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1fa94f2_124e_41c0_939d_c06a0c2aec99.slice/crio-e7659001bdc5f605fcf3f35b5838ed7580a83bbc313589705704a40e5f9f763c WatchSource:0}: Error finding container e7659001bdc5f605fcf3f35b5838ed7580a83bbc313589705704a40e5f9f763c: Status 404 returned error can't find the container with id e7659001bdc5f605fcf3f35b5838ed7580a83bbc313589705704a40e5f9f763c Oct 30 04:52:20.629491 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:20.629454 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:21.132985 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:21.132948 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" event={"ID":"e1fa94f2-124e-41c0-939d-c06a0c2aec99","Type":"ContainerStarted","Data":"07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954"} Oct 30 04:52:21.133170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:21.133004 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:21.133170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:21.133023 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" event={"ID":"e1fa94f2-124e-41c0-939d-c06a0c2aec99","Type":"ContainerStarted","Data":"e7659001bdc5f605fcf3f35b5838ed7580a83bbc313589705704a40e5f9f763c"} Oct 30 04:52:21.179177 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:21.179117 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" podStartSLOduration=2.179104229 podStartE2EDuration="2.179104229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:21.175561042 +0000 UTC m=+3264.461435818" watchObservedRunningTime="2026-10-30 04:52:21.179104229 +0000 UTC m=+3264.464979003" Oct 30 04:52:21.249847 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:21.249813 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:21.474272 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:21.474199 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:21.801965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:21.801882 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:22.018144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:22.018105 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:22.441863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:22.441823 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:22.756373 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:22.756281 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:24.188304 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:24.188253 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:24.418953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:24.418919 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:24.963080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:24.963040 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts"] Oct 30 04:52:24.967932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:24.967912 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:24.997614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:24.997588 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts"] Oct 30 04:52:25.060494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.060464 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e58a7bb-c456-4636-b455-b331d9185f70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.060684 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.060552 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h9tn\" (UniqueName: \"kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-kube-api-access-7h9tn\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.060684 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.060665 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.060767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.060700 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.060767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.060760 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.060878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.060862 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.060944 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.060926 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e58a7bb-c456-4636-b455-b331d9185f70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.061074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.061057 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.061121 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.061108 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.161993 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.161958 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.162156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.162021 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e58a7bb-c456-4636-b455-b331d9185f70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.162156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.162114 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.162156 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.162152 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.162352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.162199 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e58a7bb-c456-4636-b455-b331d9185f70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.162352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.162259 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h9tn\" (UniqueName: \"kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-kube-api-access-7h9tn\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.162352 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.162324 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.162532 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.162352 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.162532 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.162382 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.163089 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.163062 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.164264 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.163993 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.164264 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.164035 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.164492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.164110 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e58a7bb-c456-4636-b455-b331d9185f70-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.164492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.164214 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.167466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.166350 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.167466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.166866 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e58a7bb-c456-4636-b455-b331d9185f70-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.200972 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.200933 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h9tn\" (UniqueName: \"kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-kube-api-access-7h9tn\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.201915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.201885 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-v45ts\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.285601 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.285517 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:25.316875 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.316797 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:25.755953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.755921 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts"] Oct 30 04:52:25.758407 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:52:25.758364 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e58a7bb_c456_4636_b455_b331d9185f70.slice/crio-14a9d6b999722aaa572cc32b4fba9813cd4c843afd9dd39efb4e8a749c7b2177 WatchSource:0}: Error finding container 14a9d6b999722aaa572cc32b4fba9813cd4c843afd9dd39efb4e8a749c7b2177: Status 404 returned error can't find the container with id 14a9d6b999722aaa572cc32b4fba9813cd4c843afd9dd39efb4e8a749c7b2177 Oct 30 04:52:25.761625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.761587 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:25.761711 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.761668 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:25.761711 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.761701 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:25.851499 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:25.851391 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:26.050648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:26.050564 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:26.264273 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:26.264227 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" event={"ID":"7e58a7bb-c456-4636-b455-b331d9185f70","Type":"ContainerStarted","Data":"df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f"} Oct 30 04:52:26.264707 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:26.264308 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" event={"ID":"7e58a7bb-c456-4636-b455-b331d9185f70","Type":"ContainerStarted","Data":"14a9d6b999722aaa572cc32b4fba9813cd4c843afd9dd39efb4e8a749c7b2177"} Oct 30 04:52:26.285681 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:26.285649 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:26.287516 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:26.287488 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-v45ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 04:52:26.287611 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:26.287576 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 04:52:26.334799 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:26.334695 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podStartSLOduration=2.334676561 podStartE2EDuration="2.334676561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:26.327316854 +0000 UTC m=+3269.613191655" watchObservedRunningTime="2026-10-30 04:52:26.334676561 +0000 UTC m=+3269.620551341" Oct 30 04:52:26.496374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:26.496318 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:26.683562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:26.683524 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf"] Oct 30 04:52:26.691909 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:26.691881 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:26.957500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:26.954218 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:27.287276 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:27.287171 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-v45ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 04:52:27.315545 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:27.315498 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 04:52:27.346506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:27.346469 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:27.586495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:27.583543 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:27.801821 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:27.801783 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:28.112659 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:28.112632 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:52:28.286439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:28.286107 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-v45ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 04:52:28.286439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:28.286180 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 04:52:28.397386 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:28.397285 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:28.603755 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:28.603630 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:29.285941 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:29.285843 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-v45ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 04:52:29.285941 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:29.285901 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 04:52:29.498828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:29.498772 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:30.285857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:30.285818 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-v45ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 04:52:30.286084 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:30.285885 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 04:52:30.440471 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:30.440394 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:30.657497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:30.657247 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:30.889531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:30.888820 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:31.234810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:31.228354 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:31.308444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:31.307051 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-v45ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 04:52:31.308444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:31.307127 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 04:52:31.685489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:31.685288 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" podUID="e1fa94f2-124e-41c0-939d-c06a0c2aec99" containerName="payload-processing" containerID="cri-o://07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954" gracePeriod=25 Oct 30 04:52:32.029521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.029497 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:32.105990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.105022 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-plugins-config-volume\") pod \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " Oct 30 04:52:32.105990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.105111 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-routing-overlay-volume\") pod \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " Oct 30 04:52:32.105990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.105181 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-provider-credentials\") pod \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " Oct 30 04:52:32.105990 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.105227 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-kube-api-access-fdhxg\" (UniqueName: \"kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-kube-api-access-fdhxg\") pod \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\" (UID: \"e1fa94f2-124e-41c0-939d-c06a0c2aec99\") " Oct 30 04:52:32.106703 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.106662 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-routing-overlay-volume" pod "e1fa94f2-124e-41c0-939d-c06a0c2aec99" (UID: "e1fa94f2-124e-41c0-939d-c06a0c2aec99"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:52:32.107057 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.107030 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-plugins-config-volume" pod "e1fa94f2-124e-41c0-939d-c06a0c2aec99" (UID: "e1fa94f2-124e-41c0-939d-c06a0c2aec99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:52:32.110956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.110853 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-provider-credentials" pod "e1fa94f2-124e-41c0-939d-c06a0c2aec99" (UID: "e1fa94f2-124e-41c0-939d-c06a0c2aec99"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:52:32.111313 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.111273 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-kube-api-access-fdhxg" pod "e1fa94f2-124e-41c0-939d-c06a0c2aec99" (UID: "e1fa94f2-124e-41c0-939d-c06a0c2aec99"). InnerVolumeSpecName "kube-api-access-fdhxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:52:32.207663 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.207635 4793 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-routing-overlay-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:52:32.207850 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.207838 4793 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-provider-credentials\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:52:32.207920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.207912 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdhxg\" (UniqueName: \"kubernetes.io/projected/e1fa94f2-124e-41c0-939d-c06a0c2aec99-kube-api-access-fdhxg\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:52:32.207973 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.207966 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1fa94f2-124e-41c0-939d-c06a0c2aec99-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:52:32.291509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.288886 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-v45ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 04:52:32.291509 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.289005 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 04:52:32.316944 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.316905 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:32.412250 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.412212 4793 generic.go:356] "Generic (PLEG): container finished" podID="e1fa94f2-124e-41c0-939d-c06a0c2aec99" containerID="07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954" exitCode=0 Oct 30 04:52:32.412475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.412279 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" Oct 30 04:52:32.412475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.412297 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" event={"ID":"e1fa94f2-124e-41c0-939d-c06a0c2aec99","Type":"ContainerDied","Data":"07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954"} Oct 30 04:52:32.412475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.412381 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf" event={"ID":"e1fa94f2-124e-41c0-939d-c06a0c2aec99","Type":"ContainerDied","Data":"e7659001bdc5f605fcf3f35b5838ed7580a83bbc313589705704a40e5f9f763c"} Oct 30 04:52:32.412475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.412411 4793 scope.go:122] "RemoveContainer" containerID="07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954" Oct 30 04:52:32.423796 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.423780 4793 scope.go:122] "RemoveContainer" containerID="07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954" Oct 30 04:52:32.424065 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:52:32.424043 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954\": container with ID starting with 07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954 not found: ID does not exist" containerID="07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954" Oct 30 04:52:32.424131 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.424076 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954"} err="failed to get container status \"07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954\": rpc error: code = NotFound desc = could not find container \"07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954\": container with ID starting with 07bbdc098fd7f58fe64e36095088c07caebb9845ed1e6f0caef90a1f4c78c954 not found: ID does not exist" Oct 30 04:52:32.470652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.469734 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf"] Oct 30 04:52:32.476614 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.476581 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6db5744bf9-qv2bf"] Oct 30 04:52:32.655315 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.655229 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b"] Oct 30 04:52:32.656365 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.656331 4793 cpu_manager.go:414] "containerMap: removing container" podUID="e1fa94f2-124e-41c0-939d-c06a0c2aec99" containerName="payload-processing" Oct 30 04:52:32.656365 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.656360 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1fa94f2-124e-41c0-939d-c06a0c2aec99" containerName="payload-processing" Oct 30 04:52:32.656637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.656619 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1fa94f2-124e-41c0-939d-c06a0c2aec99" containerName="payload-processing" Oct 30 04:52:32.662277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.662255 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.679047 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.679024 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-e070ea-openshift-default-dockercfg-9bq8n\"" Oct 30 04:52:32.685256 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.685234 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b"] Oct 30 04:52:32.721709 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.721650 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cb982d-6448-409d-abdd-a6aa475760fd-istiod-ca-cert\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.722166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.721724 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-certs\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.722166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.721763 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-istio-token\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.722166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.721852 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cb982d-6448-409d-abdd-a6aa475760fd-istio-podinfo\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.722166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.721951 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-socket\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.722166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.722023 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-data\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.722166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.722050 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58phc\" (UniqueName: \"kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-kube-api-access-58phc\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.722166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.722088 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-envoy\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.722166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.722117 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-credential-socket\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.827470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.823633 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-certs\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.827470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.823689 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-istio-token\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.827470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.823777 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cb982d-6448-409d-abdd-a6aa475760fd-istio-podinfo\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.827470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.823862 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-socket\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.827470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.823925 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-data\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.827470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.823948 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58phc\" (UniqueName: \"kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-kube-api-access-58phc\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.827470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.823980 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-envoy\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.827470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.824006 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-credential-socket\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.827470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.824153 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cb982d-6448-409d-abdd-a6aa475760fd-istiod-ca-cert\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.827470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.824965 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cb982d-6448-409d-abdd-a6aa475760fd-istiod-ca-cert\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.827470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.826309 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-certs\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.827470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.826906 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-data\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.828279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.827936 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-socket\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.828279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.828164 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-credential-socket\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.831053 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.831024 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cb982d-6448-409d-abdd-a6aa475760fd-istio-podinfo\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.841384 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.841325 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-envoy\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.852683 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.852653 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58phc\" (UniqueName: \"kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-kube-api-access-58phc\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.856101 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.856072 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1fa94f2-124e-41c0-939d-c06a0c2aec99" path="/var/lib/kubelet/pods/e1fa94f2-124e-41c0-939d-c06a0c2aec99/volumes" Oct 30 04:52:32.857194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.857170 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-istio-token\") pod \"e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:32.984181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:32.983697 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:33.286793 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:33.286714 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-v45ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 04:52:33.286943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:33.286779 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 04:52:33.362237 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:52:33.362195 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cb982d_6448_409d_abdd_a6aa475760fd.slice/crio-4d01cbf8dc1ff5ffc2890c3ba641cc138532a37cfa864cb6c976b4118b8c31a3 WatchSource:0}: Error finding container 4d01cbf8dc1ff5ffc2890c3ba641cc138532a37cfa864cb6c976b4118b8c31a3: Status 404 returned error can't find the container with id 4d01cbf8dc1ff5ffc2890c3ba641cc138532a37cfa864cb6c976b4118b8c31a3 Oct 30 04:52:33.364527 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:33.364500 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b"] Oct 30 04:52:33.365133 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:33.365099 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:33.365238 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:33.365184 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:33.365238 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:33.365232 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:33.423812 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:33.423782 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" event={"ID":"61cb982d-6448-409d-abdd-a6aa475760fd","Type":"ContainerStarted","Data":"4d01cbf8dc1ff5ffc2890c3ba641cc138532a37cfa864cb6c976b4118b8c31a3"} Oct 30 04:52:34.109116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:34.109080 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:34.291522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:34.289785 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-v45ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 04:52:34.291522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:34.289887 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 04:52:34.436036 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:34.435999 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" event={"ID":"61cb982d-6448-409d-abdd-a6aa475760fd","Type":"ContainerStarted","Data":"39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc"} Oct 30 04:52:34.516262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:34.516207 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" podStartSLOduration=2.516191306 podStartE2EDuration="2.516191306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:34.501918393 +0000 UTC m=+3277.787793169" watchObservedRunningTime="2026-10-30 04:52:34.516191306 +0000 UTC m=+3277.802066142" Oct 30 04:52:34.987809 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:34.987775 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:34.988179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:34.988155 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 04:52:34.988354 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:34.988328 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" podUID="61cb982d-6448-409d-abdd-a6aa475760fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 04:52:35.176441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.174483 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:35.384490 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.383870 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:35.807450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.806438 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:35.813973 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.809600 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp"] Oct 30 04:52:35.817110 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.815225 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:35.824461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.821580 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp"] Oct 30 04:52:35.833579 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.833555 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"agc-cred-fe87557b739c\"" Oct 30 04:52:35.845377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.845350 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"payload-processing-external-model-e2e--a5b23dec-dockercfg-cpjvv\"" Oct 30 04:52:35.845567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.845431 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"agc-cred-0e012fe878e9\"" Oct 30 04:52:35.845696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.845677 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"openshift-service-ca.crt\"" Oct 30 04:52:35.845763 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.845750 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"payload-processing-external-model-plugins-e2e-extmod-b-91705f\"" Oct 30 04:52:35.845834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.845824 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"routing-overlay\"" Oct 30 04:52:35.845937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.845922 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-91705f\"/\"kube-root-ca.crt\"" Oct 30 04:52:35.923704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.923672 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:35.923910 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.923751 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmhsf\" (UniqueName: \"kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-kube-api-access-nmhsf\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:35.924044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.924022 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:35.924115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.924095 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:35.985958 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.985924 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 04:52:35.986148 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:35.985981 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" podUID="61cb982d-6448-409d-abdd-a6aa475760fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 04:52:36.025388 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.025357 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:36.025557 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.025405 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:36.025557 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.025501 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:36.025658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.025583 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhsf\" (UniqueName: \"kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-kube-api-access-nmhsf\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:36.026215 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.026189 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:36.026327 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.026314 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:36.028412 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.028383 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:36.049372 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.049346 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmhsf\" (UniqueName: \"kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-kube-api-access-nmhsf\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:36.171495 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.158615 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:36.290447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.290340 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-v45ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:52:36.290891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.290437 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:52:36.505888 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:52:36.505828 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b3b251_6404_4475_9a75_30e838a3dce1.slice/crio-975055471ac39c8cb149da2b38a8da1428a610211797ab429a55323915b267f3 WatchSource:0}: Error finding container 975055471ac39c8cb149da2b38a8da1428a610211797ab429a55323915b267f3: Status 404 returned error can't find the container with id 975055471ac39c8cb149da2b38a8da1428a610211797ab429a55323915b267f3 Oct 30 04:52:36.510403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.510082 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp"] Oct 30 04:52:36.989449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.988830 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 04:52:36.989449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:36.988908 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" podUID="61cb982d-6448-409d-abdd-a6aa475760fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 04:52:37.339179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:37.338516 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-v45ts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:52:37.339179 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:37.338576 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:52:37.496444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:37.495777 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:37.499332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:37.497351 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" event={"ID":"c9b3b251-6404-4475-9a75-30e838a3dce1","Type":"ContainerStarted","Data":"2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8"} Oct 30 04:52:37.499332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:37.497440 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" event={"ID":"c9b3b251-6404-4475-9a75-30e838a3dce1","Type":"ContainerStarted","Data":"975055471ac39c8cb149da2b38a8da1428a610211797ab429a55323915b267f3"} Oct 30 04:52:37.499332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:37.498599 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:37.544706 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:37.544649 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" podStartSLOduration=2.54463296 podStartE2EDuration="2.54463296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:37.54201819 +0000 UTC m=+3280.827892964" watchObservedRunningTime="2026-10-30 04:52:37.54463296 +0000 UTC m=+3280.830507736" Oct 30 04:52:37.986440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:37.985538 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 04:52:37.986440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:37.985596 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" podUID="61cb982d-6448-409d-abdd-a6aa475760fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 04:52:38.192621 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:38.192583 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:38.193030 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:38.193011 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:38.193874 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:38.193857 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 04:52:38.502255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:38.502221 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:38.702756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:38.702716 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:38.991088 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:38.991054 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 04:52:38.991500 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:38.991470 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" podUID="61cb982d-6448-409d-abdd-a6aa475760fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 04:52:39.072556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:39.072444 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:39.322194 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:39.322111 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:39.545961 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:39.545754 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:39.957627 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:39.957567 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:40.033716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.033661 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn"] Oct 30 04:52:40.047771 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.046792 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.058430 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.058385 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn"] Oct 30 04:52:40.063833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.060660 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.063833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.060690 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.063833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.060719 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.063833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.060823 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.063833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.060847 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec548c5-3019-48bc-9f48-e5894fb16bae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.063833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.060888 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.063833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.060974 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.063833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.061038 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.063833 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.061142 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8qg6\" (UniqueName: \"kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-kube-api-access-k8qg6\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.150454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.148012 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:40.150454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.149499 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:40.150714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.150673 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.171991 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8qg6\" (UniqueName: \"kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-kube-api-access-k8qg6\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.172207 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.172236 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.172268 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.172362 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.172389 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec548c5-3019-48bc-9f48-e5894fb16bae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.172506 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.172547 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.172575 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.173671 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec548c5-3019-48bc-9f48-e5894fb16bae-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.173542 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.174205 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.174727 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.176074 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.175087 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.177908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.177886 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.191967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.191937 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.207556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.197802 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8qg6\" (UniqueName: \"kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-kube-api-access-k8qg6\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.207556 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.203526 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-79qpn\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.292884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.292790 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:40.377149 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.377119 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:40.500765 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.500732 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:40.804974 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.804918 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:40.805578 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.805523 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:40.805771 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.805746 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:40.952698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.952671 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn"] Oct 30 04:52:40.974780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:40.974507 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:41.230437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:41.226662 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:41.288169 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:52:41.288030 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:52:41.486454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:41.485166 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:41.655854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:41.655699 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" event={"ID":"3ec548c5-3019-48bc-9f48-e5894fb16bae","Type":"ContainerStarted","Data":"95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9"} Oct 30 04:52:41.655854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:41.655761 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" event={"ID":"3ec548c5-3019-48bc-9f48-e5894fb16bae","Type":"ContainerStarted","Data":"d7fa63409c3e58844a2ab6d827c8edaae4bda9fc52b42275d25a66bd92626241"} Oct 30 04:52:41.770504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:41.750569 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:41.797737 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:41.785345 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" podStartSLOduration=2.785322496 podStartE2EDuration="2.785322496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:41.753305723 +0000 UTC m=+3285.039180498" watchObservedRunningTime="2026-10-30 04:52:41.785322496 +0000 UTC m=+3285.071197272" Oct 30 04:52:42.023447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:42.018172 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:42.398504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:42.394516 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:42.398504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:42.394657 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-79qpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 04:52:42.398504 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:42.394704 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 04:52:43.111449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.111081 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz"] Oct 30 04:52:43.125486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.123998 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.132195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.131918 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istiod-ca-cert\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.132195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.131999 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-podinfo\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.132195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.132061 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-socket\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.132195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.132155 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-certs\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.132195 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.132195 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-data\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.132717 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.132268 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-credential-socket\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.132717 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.132369 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-envoy\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.132717 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.132411 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjdrp\" (UniqueName: \"kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-kube-api-access-vjdrp\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.132717 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.132478 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-token\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.133394 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.133357 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz"] Oct 30 04:52:43.133635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.133608 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:43.150552 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.149087 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ed8266-openshift-default-dockercfg-cmnbn\"" Oct 30 04:52:43.234173 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.234130 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istiod-ca-cert\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.234380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.234218 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-podinfo\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.234380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.234281 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-socket\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.234380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.234375 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-certs\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.234582 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.234430 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-data\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.234582 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.234502 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-credential-socket\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.234696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.234603 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-envoy\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.234696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.234645 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdrp\" (UniqueName: \"kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-kube-api-access-vjdrp\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.234812 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.234696 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-token\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.234905 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.234881 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istiod-ca-cert\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.235114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.235096 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-data\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.235392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.235368 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-credential-socket\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.235969 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.235945 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-socket\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.236058 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.235950 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-certs\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.237842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.237818 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-envoy\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.246466 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.246441 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-podinfo\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.313449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.305611 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-token\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.327511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.327478 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdrp\" (UniqueName: \"kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-kube-api-access-vjdrp\") pod \"e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.348297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.348251 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:43.378973 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.378885 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-79qpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 04:52:43.379256 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.379224 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 04:52:43.446775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.446748 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:43.579594 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.579559 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:43.832144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.832100 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:43.948182 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:52:43.948144 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f0f764_1aa5_4d4e_a4ed_a3b61e2bd6b2.slice/crio-a6205e176bd10f7f0c838e2d0d9bf5264f5eb84e8e6130106ef00714f665fccd WatchSource:0}: Error finding container a6205e176bd10f7f0c838e2d0d9bf5264f5eb84e8e6130106ef00714f665fccd: Status 404 returned error can't find the container with id a6205e176bd10f7f0c838e2d0d9bf5264f5eb84e8e6130106ef00714f665fccd Oct 30 04:52:43.951445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.951253 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz"] Oct 30 04:52:43.952094 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.952049 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:43.952328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.952226 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:43.952328 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:43.952292 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:44.052172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:44.052140 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:44.380443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:44.379786 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-79qpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 04:52:44.380443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:44.379850 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 04:52:44.850761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:44.850725 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" event={"ID":"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2","Type":"ContainerStarted","Data":"65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e"} Oct 30 04:52:44.850936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:44.850780 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" event={"ID":"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2","Type":"ContainerStarted","Data":"a6205e176bd10f7f0c838e2d0d9bf5264f5eb84e8e6130106ef00714f665fccd"} Oct 30 04:52:44.930188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:44.930135 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" podStartSLOduration=1.9301210229999999 podStartE2EDuration="1.930121023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:52:44.915012365 +0000 UTC m=+3288.200887142" watchObservedRunningTime="2026-10-30 04:52:44.930121023 +0000 UTC m=+3288.215995798" Oct 30 04:52:45.326079 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:45.326042 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:45.377771 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:45.377730 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-79qpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 04:52:45.378043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:45.378014 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 04:52:45.450316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:45.450286 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:45.458197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:45.457881 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 04:52:45.458197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:45.457938 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" podUID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 04:52:45.545445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:45.541461 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:46.397453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:46.379892 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-79qpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 04:52:46.397453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:46.379965 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 04:52:46.463522 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:46.463486 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 04:52:46.464116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:46.464082 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" podUID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 04:52:47.306442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:47.292540 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:47.380447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:47.379626 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-79qpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 04:52:47.380447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:47.379684 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 04:52:47.478450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:47.476697 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 04:52:47.478450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:47.476774 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" podUID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 04:52:47.518163 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:47.518129 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:47.734810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:47.734737 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:48.382443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:48.378765 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-79qpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 04:52:48.382443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:48.378829 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 04:52:48.448259 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:48.448218 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 04:52:48.448441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:48.448287 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" podUID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 04:52:48.948820 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:48.947558 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:49.389529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:49.378816 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-79qpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 04:52:49.389529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:49.378882 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 04:52:49.447110 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:49.447076 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 04:52:49.447279 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:49.447130 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" podUID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 04:52:49.606732 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:49.605609 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:52:50.388127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:50.387665 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-79qpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 04:52:50.388127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:50.387731 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 04:52:50.448925 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:50.448588 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body= Oct 30 04:52:50.448925 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:50.448667 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" podUID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" Oct 30 04:52:50.622767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:50.622534 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:51.463763 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:51.463721 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:51.466021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:51.465998 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:51.467892 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:51.467873 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 04:52:51.655997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:51.655240 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:51.656390 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:51.656360 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:51.657308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:51.657287 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:52:52.281111 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:52.281028 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:54.099139 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:54.099085 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:54.161652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:54.161616 4793 generic.go:356] "Generic (PLEG): container finished" podID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerID="59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921" exitCode=0 Oct 30 04:52:54.162011 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:54.161988 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" event={"ID":"ba822078-a0dd-462f-bd7d-add7a00496b1","Type":"ContainerDied","Data":"59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921"} Oct 30 04:52:54.162949 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:54.162925 4793 scope.go:122] "RemoveContainer" containerID="59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921" Oct 30 04:52:54.205288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:54.204227 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:54.205288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:54.204339 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:54.205288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:54.204390 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:52:54.423038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:54.422876 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:54.756065 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:54.756027 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:55.206659 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:55.206618 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" event={"ID":"ba822078-a0dd-462f-bd7d-add7a00496b1","Type":"ContainerStarted","Data":"1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63"} Oct 30 04:52:55.308306 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:55.308271 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:55.457102 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:55.457017 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:52:55.457526 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:55.457497 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 04:52:55.457774 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:55.457750 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 04:52:55.522078 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:55.521997 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:56.022098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:56.021983 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:56.445553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:56.445519 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 04:52:56.445938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:56.445571 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 04:52:56.713473 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:56.713351 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:57.445608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:57.445577 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 04:52:57.445999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:57.445628 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 04:52:57.898307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:57.898217 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:58.445475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:58.445410 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 04:52:58.445658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:58.445493 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 04:52:58.491440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:58.491391 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:52:58.823756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:58.823617 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:59.034340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:59.034307 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:59.283648 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:59.283614 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:59.445207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:59.445168 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 04:52:59.445380 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:59.445233 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 04:52:59.489292 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:59.489251 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:59.696200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:59.696169 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:52:59.914041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:52:59.914003 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:00.445644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:00.445613 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body= Oct 30 04:53:00.445837 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:00.445663 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" Oct 30 04:53:01.451028 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:01.450994 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:53:01.452136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:01.452117 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 04:53:01.571093 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:01.571021 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:02.914878 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:02.914815 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:03.202445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:03.202104 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:03.416387 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:03.416353 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:04.929223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:04.929193 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:06.205581 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:06.205545 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:06.402554 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:06.402516 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:06.676834 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:06.676792 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:08.298295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:08.298258 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:09.993188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:09.993148 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:11.220696 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:53:11.220654 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:53:11.622206 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:11.622101 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:13.400398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:13.400358 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:13.604974 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:13.604913 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:13.875735 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:13.875690 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:14.226147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.226118 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq"] Oct 30 04:53:14.231072 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.231039 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.256851 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.256823 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq"] Oct 30 04:53:14.259767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.259745 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-134dad-openshift-default-dockercfg-7g8vk\"" Oct 30 04:53:14.282667 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.282633 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9h9n\" (UniqueName: \"kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-kube-api-access-r9h9n\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.282813 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.282786 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac37ac34-46aa-4ad9-889b-117548e46735-istiod-ca-cert\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.282880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.282813 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-istio-token\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.282880 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.282839 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-certs\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.282998 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.282975 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-data\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.283098 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.283082 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-socket\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.283174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.283154 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-envoy\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.283237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.283191 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac37ac34-46aa-4ad9-889b-117548e46735-istio-podinfo\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.283237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.283226 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-credential-socket\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.388547 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.388500 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9h9n\" (UniqueName: \"kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-kube-api-access-r9h9n\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.388915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.388894 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac37ac34-46aa-4ad9-889b-117548e46735-istiod-ca-cert\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.389060 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.389042 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-istio-token\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.389197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.389182 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-certs\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.389389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.389369 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-data\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.389660 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.389642 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-socket\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.389819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.389801 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-envoy\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.389966 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.389947 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac37ac34-46aa-4ad9-889b-117548e46735-istio-podinfo\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.390096 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.390080 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-credential-socket\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.402441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.390766 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-credential-socket\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.402951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.399614 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac37ac34-46aa-4ad9-889b-117548e46735-istiod-ca-cert\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.403062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.400115 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-certs\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.403161 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.400334 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-data\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.403253 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.400600 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-socket\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.420605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.420574 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac37ac34-46aa-4ad9-889b-117548e46735-istio-podinfo\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.432080 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.432048 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:14.437439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.433678 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-envoy\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.466974 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.466466 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9h9n\" (UniqueName: \"kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-kube-api-access-r9h9n\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.466974 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.466869 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-istio-token\") pod \"e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.554862 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.554717 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:14.638002 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.637966 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:14.934165 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:53:14.934132 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac37ac34_46aa_4ad9_889b_117548e46735.slice/crio-eb24019992deba55b04e1ff9200d2c5bb95bee33954309f40175746808061e5e WatchSource:0}: Error finding container eb24019992deba55b04e1ff9200d2c5bb95bee33954309f40175746808061e5e: Status 404 returned error can't find the container with id eb24019992deba55b04e1ff9200d2c5bb95bee33954309f40175746808061e5e Oct 30 04:53:14.934314 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.934195 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq"] Oct 30 04:53:14.936842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.936810 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:53:14.936959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.936873 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:53:14.936959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:14.936907 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:53:15.190511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:15.189558 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:15.433449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:15.433387 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" event={"ID":"ac37ac34-46aa-4ad9-889b-117548e46735","Type":"ContainerStarted","Data":"a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13"} Oct 30 04:53:15.433907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:15.433508 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" event={"ID":"ac37ac34-46aa-4ad9-889b-117548e46735","Type":"ContainerStarted","Data":"eb24019992deba55b04e1ff9200d2c5bb95bee33954309f40175746808061e5e"} Oct 30 04:53:15.482553 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:15.480034 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" podStartSLOduration=1.4800177030000001 podStartE2EDuration="1.480017703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:53:15.479594459 +0000 UTC m=+3318.765469235" watchObservedRunningTime="2026-10-30 04:53:15.480017703 +0000 UTC m=+3318.765892481" Oct 30 04:53:15.556487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:15.556459 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:16.562497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:16.562469 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:16.911730 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:16.911631 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:17.448309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:17.448264 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:17.449125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:17.449102 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:53:17.976267 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:17.976231 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:18.274613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:18.274516 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:18.827369 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:18.827335 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:19.640300 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:19.640267 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:19.910492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:19.888215 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:20.609199 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:20.609161 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:22.221165 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:22.221122 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:23.886762 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:23.886707 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:25.677021 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:25.676978 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:27.332286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:27.332246 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:28.910152 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:28.910116 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:30.614035 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:30.614001 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:32.409481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:32.408357 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:34.080950 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:34.080908 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:35.819275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:35.819243 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:37.498689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:37.497756 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:39.187714 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:39.187662 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:40.593291 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.593252 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7"] Oct 30 04:53:40.597117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.597093 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.616113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.616088 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-5f1508-serving-cert\"" Oct 30 04:53:40.616891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.616869 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7"] Oct 30 04:53:40.639051 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.639026 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-metrics-certs\") pod \"maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.639221 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.639066 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9229\" (UniqueName: \"kubernetes.io/projected/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-kube-api-access-j9229\") pod \"maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.639361 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.639335 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.639480 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.639376 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.741044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.741006 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.741044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.741056 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.741413 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.741291 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-metrics-certs\") pod \"maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.741413 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.741335 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9229\" (UniqueName: \"kubernetes.io/projected/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-kube-api-access-j9229\") pod \"maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.741933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.741897 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.743596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.743567 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-maas-api-tls\") pod \"maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.743740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.743651 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-metrics-certs\") pod \"maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.764390 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.764303 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:40.764390 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.764336 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9229\" (UniqueName: \"kubernetes.io/projected/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-kube-api-access-j9229\") pod \"maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.912710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.912631 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:40.974548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:40.974517 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:41.221287 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:41.221253 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7"] Oct 30 04:53:41.221464 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:53:41.221311 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6e51a1_7f4e_492f_92b1_b0025a43fef0.slice/crio-ef154586dc9ff5a4b9366a8cdec24d37496bab26d172d8f72b3fbdc8825f2a7d WatchSource:0}: Error finding container ef154586dc9ff5a4b9366a8cdec24d37496bab26d172d8f72b3fbdc8825f2a7d: Status 404 returned error can't find the container with id ef154586dc9ff5a4b9366a8cdec24d37496bab26d172d8f72b3fbdc8825f2a7d Oct 30 04:53:41.223061 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:41.223038 4793 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 04:53:41.224172 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:53:41.224143 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:53:41.694694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:41.694659 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" event={"ID":"6b6e51a1-7f4e-492f-92b1-b0025a43fef0","Type":"ContainerStarted","Data":"ef154586dc9ff5a4b9366a8cdec24d37496bab26d172d8f72b3fbdc8825f2a7d"} Oct 30 04:53:42.593757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:42.593716 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:42.705770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:42.705733 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" event={"ID":"6b6e51a1-7f4e-492f-92b1-b0025a43fef0","Type":"ContainerStarted","Data":"e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3"} Oct 30 04:53:42.706172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:42.705799 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:42.747910 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:42.747862 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" podStartSLOduration=2.268494928 podStartE2EDuration="2.747847047s" totalImagesPullingTime="479.352119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:53:42.746944906 +0000 UTC m=+3346.032819682" watchObservedRunningTime="2026-10-30 04:53:42.747847047 +0000 UTC m=+3346.033721826" Oct 30 04:53:44.334624 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:44.334585 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:46.104360 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:46.104322 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:47.920548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:47.920508 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:48.719908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:48.719880 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 04:53:49.612054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:49.612002 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:51.294871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:51.294829 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:53.009713 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:53.009672 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:54.751804 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:54.751764 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:56.519445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:56.519388 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:56.809381 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:56.809293 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:58.250665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:58.250625 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:53:59.898948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:53:59.898914 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:00.120113 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:00.120074 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:01.820702 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:01.820670 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:03.520762 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:03.520712 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:05.195888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:05.195847 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:06.813097 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:06.813062 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:08.424303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:08.424265 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:10.104405 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:10.104306 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:11.222195 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:54:11.222155 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:54:11.772997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:11.772964 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:13.509861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:13.509815 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:15.375202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:15.375163 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:17.025165 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:17.025127 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:18.626493 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:18.626458 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:20.305994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:20.305952 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:21.891288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:21.891248 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:23.104798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:23.104758 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:23.556434 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:23.556377 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:25.213572 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:25.213067 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:26.915449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:26.912907 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:28.813644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:28.813606 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:30.508991 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:30.508961 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:31.480228 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:31.480174 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:31.930570 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:31.930536 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:32.223607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:32.223482 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:32.446955 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:32.446918 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:32.665758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:32.665713 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:33.493640 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:33.493589 4793 patch_prober.go:29] interesting pod/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 04:54:33.494054 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:33.493675 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 04:54:34.050562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:34.050524 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:35.732305 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:35.732251 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:36.132254 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:36.132175 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:36.346943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:36.346857 4793 generic.go:356] "Generic (PLEG): container finished" podID="ac37ac34-46aa-4ad9-889b-117548e46735" containerID="a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13" exitCode=0 Oct 30 04:54:36.346943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:36.346931 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" event={"ID":"ac37ac34-46aa-4ad9-889b-117548e46735","Type":"ContainerDied","Data":"a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13"} Oct 30 04:54:36.347341 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:36.347325 4793 scope.go:122] "RemoveContainer" containerID="a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13" Oct 30 04:54:36.349247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:36.349213 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:54:36.349357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:36.349287 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:54:36.349357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:36.349327 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:54:36.423760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:36.423729 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:37.378441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:37.378004 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" event={"ID":"ac37ac34-46aa-4ad9-889b-117548e46735","Type":"ContainerStarted","Data":"ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03"} Oct 30 04:54:37.573443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:37.561826 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:54:37.573443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:37.561970 4793 patch_prober.go:29] interesting pod/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 04:54:37.573443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:37.562021 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 04:54:37.609477 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:37.609131 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:37.884086 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:37.884035 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:38.194126 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:38.194088 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:38.575441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:38.575334 4793 patch_prober.go:29] interesting pod/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 04:54:38.576016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:38.575978 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 04:54:39.309670 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:39.309626 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:39.563262 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:39.563171 4793 patch_prober.go:29] interesting pod/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 04:54:39.563563 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:39.563531 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 04:54:40.566442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:40.565588 4793 patch_prober.go:29] interesting pod/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 04:54:40.566442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:40.565648 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 04:54:41.030212 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:41.030162 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:41.222853 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:54:41.222819 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:54:41.561257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:41.560799 4793 patch_prober.go:29] interesting pod/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 04:54:41.561257 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:41.560875 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 04:54:42.556223 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:42.556181 4793 patch_prober.go:29] interesting pod/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body= Oct 30 04:54:42.556770 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:42.556251 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" Oct 30 04:54:42.743492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:42.743406 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:43.560664 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:43.560632 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:54:43.561147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:43.560959 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:54:43.561656 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:43.561640 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 04:54:44.356039 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:44.356003 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:45.976145 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:45.976107 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:47.961584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:47.961547 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:49.618700 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:49.618654 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:50.051690 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:50.051642 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:50.535644 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:50.535608 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:51.060047 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:51.060001 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:51.361286 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:51.361185 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:51.727448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:51.724862 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:53.511778 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:53.511734 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:55.282759 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:55.282723 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:55.503117 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:55.503083 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:56.936712 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:56.936647 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:57.165653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.165412 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:57.501693 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.501656 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:57.516148 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.516119 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m"] Oct 30 04:54:57.520072 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.520048 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" Oct 30 04:54:57.646444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.622766 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-82bec5-dockercfg-s2jnw\"" Oct 30 04:54:57.646704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.623561 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4jkq\" (UniqueName: \"kubernetes.io/projected/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-kube-api-access-z4jkq\") pod \"payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m\" (UID: \"c658db96-88c7-42b3-9cf3-5fcc5c45a21e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" Oct 30 04:54:57.650438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.646999 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m\" (UID: \"c658db96-88c7-42b3-9cf3-5fcc5c45a21e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" Oct 30 04:54:57.679696 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.628630 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-82bec5\"" Oct 30 04:54:57.769617 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.768051 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:54:57.816443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.783680 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m"] Oct 30 04:54:57.816443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.785096 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4jkq\" (UniqueName: \"kubernetes.io/projected/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-kube-api-access-z4jkq\") pod \"payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m\" (UID: \"c658db96-88c7-42b3-9cf3-5fcc5c45a21e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" Oct 30 04:54:57.816443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.785271 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m\" (UID: \"c658db96-88c7-42b3-9cf3-5fcc5c45a21e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" Oct 30 04:54:57.816443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.786907 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m\" (UID: \"c658db96-88c7-42b3-9cf3-5fcc5c45a21e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" Oct 30 04:54:57.842944 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.842876 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4jkq\" (UniqueName: \"kubernetes.io/projected/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-kube-api-access-z4jkq\") pod \"payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m\" (UID: \"c658db96-88c7-42b3-9cf3-5fcc5c45a21e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" Oct 30 04:54:57.929815 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:57.929786 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" Oct 30 04:54:58.238540 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:58.238499 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m"] Oct 30 04:54:58.238925 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:54:58.238658 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc658db96_88c7_42b3_9cf3_5fcc5c45a21e.slice/crio-29e543232b5cfeb52bafeae12c0566ee03bb68ee1806c2142effc530f37bd36c WatchSource:0}: Error finding container 29e543232b5cfeb52bafeae12c0566ee03bb68ee1806c2142effc530f37bd36c: Status 404 returned error can't find the container with id 29e543232b5cfeb52bafeae12c0566ee03bb68ee1806c2142effc530f37bd36c Oct 30 04:54:58.864324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:58.864288 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" event={"ID":"c658db96-88c7-42b3-9cf3-5fcc5c45a21e","Type":"ContainerStarted","Data":"ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7"} Oct 30 04:54:58.864525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:58.864345 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" event={"ID":"c658db96-88c7-42b3-9cf3-5fcc5c45a21e","Type":"ContainerStarted","Data":"29e543232b5cfeb52bafeae12c0566ee03bb68ee1806c2142effc530f37bd36c"} Oct 30 04:54:58.864525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:58.864368 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" Oct 30 04:54:58.906848 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:58.906791 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" podStartSLOduration=1.906777347 podStartE2EDuration="1.906777347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:54:58.902903934 +0000 UTC m=+3422.188778746" watchObservedRunningTime="2026-10-30 04:54:58.906777347 +0000 UTC m=+3422.192652123" Oct 30 04:54:59.030717 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:54:59.030681 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:00.696920 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:00.696881 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:02.398752 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:02.398716 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:04.008930 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:04.008895 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:04.984639 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:04.984603 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs"] Oct 30 04:55:05.716996 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:05.716964 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:07.531431 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:07.531384 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:09.092087 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:09.092032 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:09.876706 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:09.876679 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" Oct 30 04:55:09.985512 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:09.985468 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" podUID="78dbe08d-e4b5-42b7-9438-055fd6504bff" containerName="payload-processing" containerID="cri-o://c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c" gracePeriod=25 Oct 30 04:55:10.284677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.284643 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:55:10.378849 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.378820 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-routing-overlay-volume\") pod \"78dbe08d-e4b5-42b7-9438-055fd6504bff\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " Oct 30 04:55:10.379092 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.379048 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-plugins-config-volume\") pod \"78dbe08d-e4b5-42b7-9438-055fd6504bff\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " Oct 30 04:55:10.379240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.379225 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-kube-api-access-sqjjl\" (UniqueName: \"kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-kube-api-access-sqjjl\") pod \"78dbe08d-e4b5-42b7-9438-055fd6504bff\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " Oct 30 04:55:10.379308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.379242 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-routing-overlay-volume" pod "78dbe08d-e4b5-42b7-9438-055fd6504bff" (UID: "78dbe08d-e4b5-42b7-9438-055fd6504bff"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:55:10.379308 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.379299 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-provider-credentials\" (UniqueName: \"kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-provider-credentials\") pod \"78dbe08d-e4b5-42b7-9438-055fd6504bff\" (UID: \"78dbe08d-e4b5-42b7-9438-055fd6504bff\") " Oct 30 04:55:10.380174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.379721 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-plugins-config-volume" pod "78dbe08d-e4b5-42b7-9438-055fd6504bff" (UID: "78dbe08d-e4b5-42b7-9438-055fd6504bff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:55:10.380174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.379977 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:55:10.380174 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.379999 4793 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/78dbe08d-e4b5-42b7-9438-055fd6504bff-routing-overlay-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:55:10.382389 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.382366 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-provider-credentials" pod "78dbe08d-e4b5-42b7-9438-055fd6504bff" (UID: "78dbe08d-e4b5-42b7-9438-055fd6504bff"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:55:10.391572 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.391539 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-kube-api-access-sqjjl" pod "78dbe08d-e4b5-42b7-9438-055fd6504bff" (UID: "78dbe08d-e4b5-42b7-9438-055fd6504bff"). InnerVolumeSpecName "kube-api-access-sqjjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:55:10.481996 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.481965 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqjjl\" (UniqueName: \"kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-kube-api-access-sqjjl\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:55:10.481996 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.481996 4793 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/78dbe08d-e4b5-42b7-9438-055fd6504bff-provider-credentials\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:55:10.826525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.826405 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:10.961894 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.961860 4793 generic.go:356] "Generic (PLEG): container finished" podID="78dbe08d-e4b5-42b7-9438-055fd6504bff" containerID="c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c" exitCode=0 Oct 30 04:55:10.962166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.961904 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" Oct 30 04:55:10.962307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.961895 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" event={"ID":"78dbe08d-e4b5-42b7-9438-055fd6504bff","Type":"ContainerDied","Data":"c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c"} Oct 30 04:55:10.962307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.962277 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs" event={"ID":"78dbe08d-e4b5-42b7-9438-055fd6504bff","Type":"ContainerDied","Data":"4ff5eebfa30d66d1d24057907322faec9cdd5951b600545e3104053a2914be2e"} Oct 30 04:55:10.962307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.962307 4793 scope.go:122] "RemoveContainer" containerID="c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c" Oct 30 04:55:10.974778 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.974756 4793 scope.go:122] "RemoveContainer" containerID="c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c" Oct 30 04:55:10.975081 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:10.975059 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c\": container with ID starting with c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c not found: ID does not exist" containerID="c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c" Oct 30 04:55:10.975166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:10.975094 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c"} err="failed to get container status \"c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c\": rpc error: code = NotFound desc = could not find container \"c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c\": container with ID starting with c3a2abf0ba6d3966279d96d05d9bd055131924ad9d796453331b821feef7e84c not found: ID does not exist" Oct 30 04:55:11.015083 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:11.015059 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs"] Oct 30 04:55:11.026738 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:11.026715 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-483bea/payload-processing-external-model-e2e-extmod-b-483bea-8448985qs"] Oct 30 04:55:11.220054 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:11.220016 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:55:12.525192 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:12.525153 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:12.854306 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:12.854147 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78dbe08d-e4b5-42b7-9438-055fd6504bff" path="/var/lib/kubelet/pods/78dbe08d-e4b5-42b7-9438-055fd6504bff/volumes" Oct 30 04:55:12.919397 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:12.919367 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-91705f/agc-cred-fe87557b739c: secret "agc-cred-fe87557b739c" not found Oct 30 04:55:12.919397 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:12.919390 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp: secret "agc-cred-fe87557b739c" not found Oct 30 04:55:12.919617 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:12.919469 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials podName:c9b3b251-6404-4475-9a75-30e838a3dce1 nodeName:}" failed. No retries permitted until 2026-10-30 04:55:13.419452962 +0000 UTC m=+3436.705327732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" (UID: "c9b3b251-6404-4475-9a75-30e838a3dce1") : secret "agc-cred-fe87557b739c" not found Oct 30 04:55:13.037443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:13.031805 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:13.249342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:13.249311 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:13.431811 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:13.431783 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-91705f/agc-cred-fe87557b739c: secret "agc-cred-fe87557b739c" not found Oct 30 04:55:13.431811 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:13.431808 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp: secret "agc-cred-fe87557b739c" not found Oct 30 04:55:13.432046 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:13.431878 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials podName:c9b3b251-6404-4475-9a75-30e838a3dce1 nodeName:}" failed. No retries permitted until 2026-10-30 04:55:14.431858708 +0000 UTC m=+3437.717733462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" (UID: "c9b3b251-6404-4475-9a75-30e838a3dce1") : secret "agc-cred-fe87557b739c" not found Oct 30 04:55:14.238973 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.238942 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf"] Oct 30 04:55:14.239798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.239772 4793 cpu_manager.go:414] "containerMap: removing container" podUID="78dbe08d-e4b5-42b7-9438-055fd6504bff" containerName="payload-processing" Oct 30 04:55:14.239798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.239793 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78dbe08d-e4b5-42b7-9438-055fd6504bff" containerName="payload-processing" Oct 30 04:55:14.239985 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.239906 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="78dbe08d-e4b5-42b7-9438-055fd6504bff" containerName="payload-processing" Oct 30 04:55:14.243288 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.243271 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.279355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.279312 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf"] Oct 30 04:55:14.352329 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.352293 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.352567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.352546 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8dzf\" (UniqueName: \"kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-kube-api-access-f8dzf\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.352782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.352761 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.352859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.352811 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.356754 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.356710 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:14.454157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.454113 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.454157 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.454160 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.454433 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.454185 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.454433 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:14.454350 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-91705f/agc-cred-0e012fe878e9: secret "agc-cred-0e012fe878e9" not found Oct 30 04:55:14.454433 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:14.454374 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf: secret "agc-cred-0e012fe878e9" not found Oct 30 04:55:14.454433 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:14.454388 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-91705f/agc-cred-0e012fe878e9: secret "agc-cred-0e012fe878e9" not found Oct 30 04:55:14.454433 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:14.454404 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-91705f/agc-cred-fe87557b739c: secret "agc-cred-fe87557b739c" not found Oct 30 04:55:14.454433 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.454405 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8dzf\" (UniqueName: \"kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-kube-api-access-f8dzf\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.454763 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:14.454431 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp: [secret "agc-cred-0e012fe878e9" not found, secret "agc-cred-fe87557b739c" not found] Oct 30 04:55:14.454763 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:14.454473 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-provider-credentials podName:8cd1746a-385d-4b12-9e79-c2757d3b79de nodeName:}" failed. No retries permitted until 2026-10-30 04:55:14.954450857 +0000 UTC m=+3438.240325628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" (UID: "8cd1746a-385d-4b12-9e79-c2757d3b79de") : secret "agc-cred-0e012fe878e9" not found Oct 30 04:55:14.454763 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:14.454529 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials podName:c9b3b251-6404-4475-9a75-30e838a3dce1 nodeName:}" failed. No retries permitted until 2026-10-30 04:55:16.454512027 +0000 UTC m=+3439.740386785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" (UID: "c9b3b251-6404-4475-9a75-30e838a3dce1") : [secret "agc-cred-0e012fe878e9" not found, secret "agc-cred-fe87557b739c" not found] Oct 30 04:55:14.454989 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.454968 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.455169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.455146 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.477668 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.477636 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8dzf\" (UniqueName: \"kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-kube-api-access-f8dzf\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.588910 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.588815 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:14.809460 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.809408 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:14.983486 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:14.983446 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:14.983761 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:14.983630 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-91705f/agc-cred-0e012fe878e9: secret "agc-cred-0e012fe878e9" not found Oct 30 04:55:14.983761 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:14.983657 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf: secret "agc-cred-0e012fe878e9" not found Oct 30 04:55:14.983761 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:14.983726 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-provider-credentials podName:8cd1746a-385d-4b12-9e79-c2757d3b79de nodeName:}" failed. No retries permitted until 2026-10-30 04:55:15.98370598 +0000 UTC m=+3439.269580748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" (UID: "8cd1746a-385d-4b12-9e79-c2757d3b79de") : secret "agc-cred-0e012fe878e9" not found Oct 30 04:55:15.022698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:15.022660 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:15.255193 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:15.255081 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:15.666128 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:15.666049 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp"] Oct 30 04:55:15.678572 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:15.678541 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf"] Oct 30 04:55:15.679254 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:15.679215 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" podUID="8cd1746a-385d-4b12-9e79-c2757d3b79de" Oct 30 04:55:15.845482 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:15.845452 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:16.006817 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.006777 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:16.007181 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:16.007162 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-91705f/agc-cred-0e012fe878e9: secret "agc-cred-0e012fe878e9" not found Oct 30 04:55:16.007305 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:16.007183 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf: secret "agc-cred-0e012fe878e9" not found Oct 30 04:55:16.007305 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:16.007252 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-provider-credentials podName:8cd1746a-385d-4b12-9e79-c2757d3b79de nodeName:}" failed. No retries permitted until 2026-10-30 04:55:18.007231851 +0000 UTC m=+3441.293106621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" (UID: "8cd1746a-385d-4b12-9e79-c2757d3b79de") : secret "agc-cred-0e012fe878e9" not found Oct 30 04:55:16.130688 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.130647 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:16.273497 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.269089 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:16.303007 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.302974 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:16.435300 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.435242 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-plugins-config-volume\") pod \"8cd1746a-385d-4b12-9e79-c2757d3b79de\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " Oct 30 04:55:16.435555 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.435468 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-routing-overlay-volume\") pod \"8cd1746a-385d-4b12-9e79-c2757d3b79de\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " Oct 30 04:55:16.435686 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.435667 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-kube-api-access-f8dzf\" (UniqueName: \"kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-kube-api-access-f8dzf\") pod \"8cd1746a-385d-4b12-9e79-c2757d3b79de\" (UID: \"8cd1746a-385d-4b12-9e79-c2757d3b79de\") " Oct 30 04:55:16.435769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.435742 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-plugins-config-volume" pod "8cd1746a-385d-4b12-9e79-c2757d3b79de" (UID: "8cd1746a-385d-4b12-9e79-c2757d3b79de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:55:16.435862 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.435844 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-routing-overlay-volume" pod "8cd1746a-385d-4b12-9e79-c2757d3b79de" (UID: "8cd1746a-385d-4b12-9e79-c2757d3b79de"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:55:16.436453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.436394 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:55:16.436453 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.436431 4793 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8cd1746a-385d-4b12-9e79-c2757d3b79de-routing-overlay-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:55:16.437676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.437652 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-kube-api-access-f8dzf" pod "8cd1746a-385d-4b12-9e79-c2757d3b79de" (UID: "8cd1746a-385d-4b12-9e79-c2757d3b79de"). InnerVolumeSpecName "kube-api-access-f8dzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:55:16.538204 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:16.538130 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-91705f/agc-cred-0e012fe878e9: secret "agc-cred-0e012fe878e9" not found Oct 30 04:55:16.538204 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:16.538154 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-91705f/agc-cred-fe87557b739c: secret "agc-cred-fe87557b739c" not found Oct 30 04:55:16.538204 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:16.538164 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp: [secret "agc-cred-0e012fe878e9" not found, secret "agc-cred-fe87557b739c" not found] Oct 30 04:55:16.538204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:16.538184 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8dzf\" (UniqueName: \"kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-kube-api-access-f8dzf\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:55:16.538478 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:16.538218 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials podName:c9b3b251-6404-4475-9a75-30e838a3dce1 nodeName:}" failed. No retries permitted until 2026-10-30 04:55:20.538203551 +0000 UTC m=+3443.824078317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" (UID: "c9b3b251-6404-4475-9a75-30e838a3dce1") : [secret "agc-cred-0e012fe878e9" not found, secret "agc-cred-fe87557b739c" not found] Oct 30 04:55:17.276959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:17.276930 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf" Oct 30 04:55:17.740511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:17.731511 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf"] Oct 30 04:55:17.769263 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:17.769222 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-7448jkrmf"] Oct 30 04:55:17.870135 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:17.870073 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:17.898758 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:17.898723 4793 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8cd1746a-385d-4b12-9e79-c2757d3b79de-provider-credentials\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:55:18.849237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:18.849198 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd1746a-385d-4b12-9e79-c2757d3b79de" path="/var/lib/kubelet/pods/8cd1746a-385d-4b12-9e79-c2757d3b79de/volumes" Oct 30 04:55:19.562740 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:19.562697 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:20.640180 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:20.640150 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-91705f/agc-cred-0e012fe878e9: secret "agc-cred-0e012fe878e9" not found Oct 30 04:55:20.640180 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:20.640175 4793 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-91705f/agc-cred-fe87557b739c: secret "agc-cred-fe87557b739c" not found Oct 30 04:55:20.640180 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:20.640184 4793 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp: [secret "agc-cred-0e012fe878e9" not found, secret "agc-cred-fe87557b739c" not found] Oct 30 04:55:20.640655 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:20.640236 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials podName:c9b3b251-6404-4475-9a75-30e838a3dce1 nodeName:}" failed. No retries permitted until 2026-10-30 04:55:28.640222188 +0000 UTC m=+3451.926096945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" (UID: "c9b3b251-6404-4475-9a75-30e838a3dce1") : [secret "agc-cred-0e012fe878e9" not found, secret "agc-cred-fe87557b739c" not found] Oct 30 04:55:20.667127 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:20.667083 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" podUID="c9b3b251-6404-4475-9a75-30e838a3dce1" containerName="payload-processing" containerID="cri-o://2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8" gracePeriod=25 Oct 30 04:55:20.960554 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:20.960526 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:55:21.045989 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.045956 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-routing-overlay-volume\") pod \"c9b3b251-6404-4475-9a75-30e838a3dce1\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " Oct 30 04:55:21.046169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.046107 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-kube-api-access-nmhsf\" (UniqueName: \"kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-kube-api-access-nmhsf\") pod \"c9b3b251-6404-4475-9a75-30e838a3dce1\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " Oct 30 04:55:21.046296 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.046268 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials\") pod \"c9b3b251-6404-4475-9a75-30e838a3dce1\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " Oct 30 04:55:21.046377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.046298 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-plugins-config-volume\") pod \"c9b3b251-6404-4475-9a75-30e838a3dce1\" (UID: \"c9b3b251-6404-4475-9a75-30e838a3dce1\") " Oct 30 04:55:21.046531 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.046380 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-routing-overlay-volume" pod "c9b3b251-6404-4475-9a75-30e838a3dce1" (UID: "c9b3b251-6404-4475-9a75-30e838a3dce1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:55:21.046932 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.046912 4793 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-routing-overlay-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:55:21.047023 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.046946 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-plugins-config-volume" pod "c9b3b251-6404-4475-9a75-30e838a3dce1" (UID: "c9b3b251-6404-4475-9a75-30e838a3dce1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:55:21.049203 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.049164 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials" pod "c9b3b251-6404-4475-9a75-30e838a3dce1" (UID: "c9b3b251-6404-4475-9a75-30e838a3dce1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:55:21.049767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.049742 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-kube-api-access-nmhsf" pod "c9b3b251-6404-4475-9a75-30e838a3dce1" (UID: "c9b3b251-6404-4475-9a75-30e838a3dce1"). InnerVolumeSpecName "kube-api-access-nmhsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:55:21.098623 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.098589 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b"] Oct 30 04:55:21.099402 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.099381 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c9b3b251-6404-4475-9a75-30e838a3dce1" containerName="payload-processing" Oct 30 04:55:21.099539 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.099403 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b3b251-6404-4475-9a75-30e838a3dce1" containerName="payload-processing" Oct 30 04:55:21.099635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.099620 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b3b251-6404-4475-9a75-30e838a3dce1" containerName="payload-processing" Oct 30 04:55:21.103590 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.103570 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.126557 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.126530 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b"] Oct 30 04:55:21.126689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.126676 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f9cd17-serving-cert\"" Oct 30 04:55:21.148716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.148685 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcnft\" (UniqueName: \"kubernetes.io/projected/3083e820-94e6-4000-8987-a57f3bd03798-kube-api-access-bcnft\") pod \"maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.149118 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.149092 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3083e820-94e6-4000-8987-a57f3bd03798-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.149246 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.149149 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-maas-api-tls\") pod \"maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.149246 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.149198 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-metrics-certs\") pod \"maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.149397 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.149379 4793 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-provider-credentials\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:55:21.149481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.149395 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9b3b251-6404-4475-9a75-30e838a3dce1-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:55:21.149481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.149409 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmhsf\" (UniqueName: \"kubernetes.io/projected/c9b3b251-6404-4475-9a75-30e838a3dce1-kube-api-access-nmhsf\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:55:21.250924 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.250842 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcnft\" (UniqueName: \"kubernetes.io/projected/3083e820-94e6-4000-8987-a57f3bd03798-kube-api-access-bcnft\") pod \"maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.251467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.251099 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3083e820-94e6-4000-8987-a57f3bd03798-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.251819 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.251408 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-maas-api-tls\") pod \"maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.251892 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.251858 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-metrics-certs\") pod \"maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.252063 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.252032 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3083e820-94e6-4000-8987-a57f3bd03798-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.254000 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.253978 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-maas-api-tls\") pod \"maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.254232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.254212 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-metrics-certs\") pod \"maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.274112 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.274089 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcnft\" (UniqueName: \"kubernetes.io/projected/3083e820-94e6-4000-8987-a57f3bd03798-kube-api-access-bcnft\") pod \"maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.291507 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.291481 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:21.319661 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.319629 4793 generic.go:356] "Generic (PLEG): container finished" podID="c9b3b251-6404-4475-9a75-30e838a3dce1" containerID="2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8" exitCode=0 Oct 30 04:55:21.319787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.319709 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" event={"ID":"c9b3b251-6404-4475-9a75-30e838a3dce1","Type":"ContainerDied","Data":"2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8"} Oct 30 04:55:21.319787 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.319764 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" event={"ID":"c9b3b251-6404-4475-9a75-30e838a3dce1","Type":"ContainerDied","Data":"975055471ac39c8cb149da2b38a8da1428a610211797ab429a55323915b267f3"} Oct 30 04:55:21.319906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.319792 4793 scope.go:122] "RemoveContainer" containerID="2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8" Oct 30 04:55:21.319906 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.319717 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp" Oct 30 04:55:21.333476 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.333454 4793 scope.go:122] "RemoveContainer" containerID="2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8" Oct 30 04:55:21.333779 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:21.333759 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8\": container with ID starting with 2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8 not found: ID does not exist" containerID="2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8" Oct 30 04:55:21.333844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.333787 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8"} err="failed to get container status \"2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8\": rpc error: code = NotFound desc = could not find container \"2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8\": container with ID starting with 2dec7f96bbefe3550263d82730ec7d99ddd08bc7e24af80f29eb770735d2a1c8 not found: ID does not exist" Oct 30 04:55:21.384291 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.384225 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp"] Oct 30 04:55:21.390763 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.390726 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-91705f/payload-processing-external-model-e2e-extmod-b-91705f-5cc4b58dp"] Oct 30 04:55:21.425470 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.418248 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:21.521020 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.520981 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:21.747297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:21.747266 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b"] Oct 30 04:55:21.747723 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:55:21.747599 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3083e820_94e6_4000_8987_a57f3bd03798.slice/crio-523a945010b76ca3d23bda3006b8f53955562f52a29dc59eba6a849b157c0e88 WatchSource:0}: Error finding container 523a945010b76ca3d23bda3006b8f53955562f52a29dc59eba6a849b157c0e88: Status 404 returned error can't find the container with id 523a945010b76ca3d23bda3006b8f53955562f52a29dc59eba6a849b157c0e88 Oct 30 04:55:22.010529 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.010494 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:22.305503 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.305462 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb"] Oct 30 04:55:22.310676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.310629 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.329035 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.329010 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb"] Oct 30 04:55:22.334710 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.334685 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-449d83-openshift-default-dockercfg-q5lq7\"" Oct 30 04:55:22.345723 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.345696 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" event={"ID":"3083e820-94e6-4000-8987-a57f3bd03798","Type":"ContainerStarted","Data":"7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af"} Oct 30 04:55:22.345839 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.345742 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" event={"ID":"3083e820-94e6-4000-8987-a57f3bd03798","Type":"ContainerStarted","Data":"523a945010b76ca3d23bda3006b8f53955562f52a29dc59eba6a849b157c0e88"} Oct 30 04:55:22.346355 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.346335 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:22.370520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.370485 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-credential-socket\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.370666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.370590 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-socket\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.370666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.370639 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istiod-ca-cert\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.370794 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.370731 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-certs\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.370871 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.370839 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-token\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.370933 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.370890 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-data\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.370991 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.370931 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs7tm\" (UniqueName: \"kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-kube-api-access-bs7tm\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.371062 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.371030 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-envoy\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.371125 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.371099 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-podinfo\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.514445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.513486 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:22.514700 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.514669 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-token\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.514767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.514712 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-data\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.514767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.514756 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs7tm\" (UniqueName: \"kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-kube-api-access-bs7tm\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.514948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.514870 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-envoy\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.514948 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.514917 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-podinfo\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.525489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.515120 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-credential-socket\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.525489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.515180 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-socket\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.525489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.515221 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istiod-ca-cert\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.525489 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.515296 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-certs\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.541442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.538899 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" podStartSLOduration=1.086054957 podStartE2EDuration="1.538876163s" totalImagesPullingTime="452.821206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:22.413607702 +0000 UTC m=+3445.699482478" watchObservedRunningTime="2026-10-30 04:55:22.538876163 +0000 UTC m=+3445.824750941" Oct 30 04:55:22.546515 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.545914 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-data\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.549450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.549114 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-envoy\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.561454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.551230 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-socket\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.561454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.551838 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-credential-socket\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.561454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.552656 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istiod-ca-cert\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.561454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.552888 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-certs\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.595816 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.588828 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-podinfo\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.623221 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.623187 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs7tm\" (UniqueName: \"kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-kube-api-access-bs7tm\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.724575 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.724473 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-token\") pod \"e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:22.732872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.732837 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:22.848438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.848342 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9b3b251-6404-4475-9a75-30e838a3dce1" path="/var/lib/kubelet/pods/c9b3b251-6404-4475-9a75-30e838a3dce1/volumes" Oct 30 04:55:22.945105 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:22.945074 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:23.181910 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:23.181871 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:23.279054 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:55:23.279024 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b6d0eb_2e34_43b0_9b83_c448f4b0fc28.slice/crio-6467e8f29ed6f3f149cf44fd6047d952dafc51b1e720dbf707f2c983649a4d41 WatchSource:0}: Error finding container 6467e8f29ed6f3f149cf44fd6047d952dafc51b1e720dbf707f2c983649a4d41: Status 404 returned error can't find the container with id 6467e8f29ed6f3f149cf44fd6047d952dafc51b1e720dbf707f2c983649a4d41 Oct 30 04:55:23.280705 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:23.280675 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb"] Oct 30 04:55:23.283487 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:23.283442 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:55:23.283596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:23.283533 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:55:23.283596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:23.283577 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:55:23.356043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:23.356014 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" event={"ID":"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28","Type":"ContainerStarted","Data":"6467e8f29ed6f3f149cf44fd6047d952dafc51b1e720dbf707f2c983649a4d41"} Oct 30 04:55:24.367649 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:24.367612 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" event={"ID":"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28","Type":"ContainerStarted","Data":"7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698"} Oct 30 04:55:24.429457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:24.429385 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" podStartSLOduration=2.429370752 podStartE2EDuration="2.429370752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:55:24.423399808 +0000 UTC m=+3447.709274582" watchObservedRunningTime="2026-10-30 04:55:24.429370752 +0000 UTC m=+3447.715245526" Oct 30 04:55:24.939802 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:24.939767 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:24.945600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:24.945579 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:24.952243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:24.952218 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:25.376445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:25.376345 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:25.377565 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:25.377528 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:55:26.630025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:26.629995 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:27.095110 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:27.095076 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:27.379320 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:27.379236 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:28.556843 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:28.556806 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:28.785178 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:28.785127 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:29.373653 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:29.373620 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 04:55:30.319988 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:30.319955 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:32.005085 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:32.005048 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:32.513385 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:32.513348 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:32.739936 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:32.739843 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:33.674676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:33.674632 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:35.330548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:35.330514 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:37.003333 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:37.003284 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:38.598220 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:38.598184 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:40.225207 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:40.225108 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:41.219279 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:55:41.219241 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:55:42.010925 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:42.010550 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:42.237318 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:42.237270 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:42.447035 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:42.447004 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:43.736567 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:43.736532 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:45.435535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:45.435457 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:47.327831 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:47.327793 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:49.095467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:49.095380 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:50.924743 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:50.924701 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:52.743329 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:52.743290 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:54.414277 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:54.414228 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:56.206296 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:56.206259 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:57.072375 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:57.072340 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:58.039405 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:58.039366 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:55:59.695782 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:55:59.695730 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:01.380468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:01.380394 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:03.131860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:03.131829 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:04.915233 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:04.915191 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:06.549440 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:06.543676 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:08.319893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:08.319855 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:10.071155 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:10.071116 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:11.220240 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:56:11.220203 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:56:11.722715 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:11.722663 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:12.424366 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:12.424329 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:12.635403 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:12.635369 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:13.264124 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:13.264085 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:13.516083 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:13.515993 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:13.734314 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:13.734284 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:14.445441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:14.439944 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:14.672398 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:14.672363 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:14.885255 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:14.885164 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:15.104866 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.104832 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:15.379601 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.379567 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf"] Oct 30 04:56:15.384947 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.384925 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:15.409905 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.408357 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-449d83-serving-cert\"" Oct 30 04:56:15.409905 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.409547 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf"] Oct 30 04:56:15.448520 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.448486 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-metrics-certs\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:15.448704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.448663 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15998a1b-c496-47e0-8887-0315b0354f2b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:15.448704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.448694 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsmd5\" (UniqueName: \"kubernetes.io/projected/15998a1b-c496-47e0-8887-0315b0354f2b-kube-api-access-xsmd5\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:15.448856 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.448722 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-maas-api-tls\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:15.550725 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.550690 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-metrics-certs\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:15.550907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.550816 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15998a1b-c496-47e0-8887-0315b0354f2b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:15.550907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.550845 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsmd5\" (UniqueName: \"kubernetes.io/projected/15998a1b-c496-47e0-8887-0315b0354f2b-kube-api-access-xsmd5\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:15.550907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.550871 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-maas-api-tls\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:15.551097 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:56:15.551002 4793 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-serving-cert: secret "maas-api-e2e-inf-a-449d83-serving-cert" not found Oct 30 04:56:15.551097 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:56:15.551069 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-maas-api-tls podName:15998a1b-c496-47e0-8887-0315b0354f2b nodeName:}" failed. No retries permitted until 2026-10-30 04:56:16.051049181 +0000 UTC m=+3499.336923953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-maas-api-tls") pod "maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" (UID: "15998a1b-c496-47e0-8887-0315b0354f2b") : secret "maas-api-e2e-inf-a-449d83-serving-cert" not found Oct 30 04:56:15.551597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.551570 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15998a1b-c496-47e0-8887-0315b0354f2b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:15.553208 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.553180 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-metrics-certs\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:15.598445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.595650 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:15.601597 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.601567 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsmd5\" (UniqueName: \"kubernetes.io/projected/15998a1b-c496-47e0-8887-0315b0354f2b-kube-api-access-xsmd5\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:15.813351 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:15.813311 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:16.059332 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:16.059301 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-maas-api-tls\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:16.061797 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:16.061768 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-maas-api-tls\") pod \"maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:16.323392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:16.323361 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:16.591635 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:16.591552 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:16.637556 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:56:16.637520 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15998a1b_c496_47e0_8887_0315b0354f2b.slice/crio-511c949dceed8807a4e033e8e379525660a2683fb59b68cb441f0450da299f7e WatchSource:0}: Error finding container 511c949dceed8807a4e033e8e379525660a2683fb59b68cb441f0450da299f7e: Status 404 returned error can't find the container with id 511c949dceed8807a4e033e8e379525660a2683fb59b68cb441f0450da299f7e Oct 30 04:56:16.638898 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:16.638861 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf"] Oct 30 04:56:16.817548 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:16.817511 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:16.895876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:16.895795 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" event={"ID":"15998a1b-c496-47e0-8887-0315b0354f2b","Type":"ContainerStarted","Data":"511c949dceed8807a4e033e8e379525660a2683fb59b68cb441f0450da299f7e"} Oct 30 04:56:17.494768 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:17.491648 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:17.905248 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:17.905160 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" event={"ID":"15998a1b-c496-47e0-8887-0315b0354f2b","Type":"ContainerStarted","Data":"88092f6fb138f236c60770bcb94adebb115a73fc097814337b8006d2a3a55605"} Oct 30 04:56:17.905689 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:17.905393 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:17.971444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:17.967389 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" podStartSLOduration=2.598081746 podStartE2EDuration="2.967369296s" totalImagesPullingTime="369.28755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:56:17.959765707 +0000 UTC m=+3501.245640484" watchObservedRunningTime="2026-10-30 04:56:17.967369296 +0000 UTC m=+3501.253244072" Oct 30 04:56:19.172349 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:19.172314 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:20.836608 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:20.836575 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:21.529674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:21.529642 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:21.747032 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:21.746996 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:22.011247 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:22.011209 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:22.466329 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:22.466299 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:22.680060 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:22.680024 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:22.890073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:22.889992 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:23.919652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:23.919623 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 04:56:24.415637 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:24.415550 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:24.779862 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:24.779816 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:25.000859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:25.000813 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:26.197451 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:26.197380 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:26.489064 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:26.488974 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:26.713018 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:26.712982 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:28.016584 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:28.016526 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:28.233974 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:28.233939 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:28.500723 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:28.497372 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:29.767248 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:29.767211 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:31.521342 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:31.521308 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:33.028438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:33.020502 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:33.251445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:33.244702 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:33.465177 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:33.465136 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:34.398760 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:34.398723 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:35.219902 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:35.219873 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:37.055596 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:37.055503 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:37.600562 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:37.600528 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:38.840197 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:38.840159 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:40.627170 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:40.627131 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:40.869625 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:40.869591 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:41.575468 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:56:41.573603 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:56:42.436356 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:42.436317 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:44.207309 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:44.207264 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:45.884781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:45.884744 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:47.545163 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:47.545122 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:49.223085 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:49.223038 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:50.882780 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:50.882742 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:52.636411 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:52.636372 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:54.411146 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:54.411110 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:56.161650 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:56.160088 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:56.382386 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:56.382338 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:56.916513 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:56.916465 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:57.197173 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:57.197136 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:57.433283 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:57.433249 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:57.903200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:57.903165 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:59.518879 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:59.518837 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:56:59.761511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:56:59.761478 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:01.252382 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:01.252338 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:02.900745 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:02.900707 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:03.465191 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:03.465156 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:03.677909 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:03.677877 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:03.932303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:03.932269 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:04.347211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.347137 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f"] Oct 30 04:57:04.351848 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.351827 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.382005 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.381976 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f"] Oct 30 04:57:04.394934 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.394909 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9f158a-openshift-default-dockercfg-qrcgk\"" Oct 30 04:57:04.472443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.460460 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65430b4d-69e5-4e30-8a47-9c1e9003759c-istiod-ca-cert\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.473697 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.473666 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-socket\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.473953 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.473921 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rspq\" (UniqueName: \"kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-kube-api-access-7rspq\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.474136 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.474105 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-token\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.474314 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.474295 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-envoy\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.474455 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.474437 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-data\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.474605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.474589 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-podinfo\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.474883 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.474864 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-certs\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.477271 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.475168 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-credential-socket\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.518893 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.518854 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:04.582524 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.582485 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-certs\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.595445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.590753 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-credential-socket\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.595445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.590925 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65430b4d-69e5-4e30-8a47-9c1e9003759c-istiod-ca-cert\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.595445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.591016 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-socket\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.595445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.591054 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rspq\" (UniqueName: \"kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-kube-api-access-7rspq\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.595445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.591101 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-token\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.595445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.591155 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-envoy\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.595445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.591180 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-data\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.595445 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.591218 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-podinfo\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.612557 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.584122 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-certs\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.613211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.613173 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-credential-socket\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.614186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.614166 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65430b4d-69e5-4e30-8a47-9c1e9003759c-istiod-ca-cert\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.614643 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.614625 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-socket\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.623457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.618728 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-data\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.623457 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.621402 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-podinfo\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.640605 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.640567 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-envoy\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.651135 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.651104 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-token\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.661606 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.658592 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rspq\" (UniqueName: \"kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-kube-api-access-7rspq\") pod \"e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.667956 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.667927 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:04.746921 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:04.746888 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:05.075067 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:05.075033 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f"] Oct 30 04:57:05.075931 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:57:05.075906 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65430b4d_69e5_4e30_8a47_9c1e9003759c.slice/crio-789438c125c67cf14fdf439da33a90985271765ee7732894772978ec6b1fc5c3 WatchSource:0}: Error finding container 789438c125c67cf14fdf439da33a90985271765ee7732894772978ec6b1fc5c3: Status 404 returned error can't find the container with id 789438c125c67cf14fdf439da33a90985271765ee7732894772978ec6b1fc5c3 Oct 30 04:57:05.079698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:05.079654 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:57:05.079806 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:05.079747 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:57:05.079806 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:05.079789 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:57:05.772666 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:05.772629 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" event={"ID":"65430b4d-69e5-4e30-8a47-9c1e9003759c","Type":"ContainerStarted","Data":"70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7"} Oct 30 04:57:05.772854 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:05.772684 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" event={"ID":"65430b4d-69e5-4e30-8a47-9c1e9003759c","Type":"ContainerStarted","Data":"789438c125c67cf14fdf439da33a90985271765ee7732894772978ec6b1fc5c3"} Oct 30 04:57:05.840259 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:05.840208 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" podStartSLOduration=1.840197182 podStartE2EDuration="1.840197182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:05.827682201 +0000 UTC m=+3549.113556977" watchObservedRunningTime="2026-10-30 04:57:05.840197182 +0000 UTC m=+3549.126071957" Oct 30 04:57:06.408558 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:06.408517 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:06.668877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:06.668758 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:06.675240 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:06.675216 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:06.781204 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:06.781165 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:06.782107 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:06.782083 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:57:08.363467 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:08.362945 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:08.596388 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:08.596344 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:09.308931 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:09.308894 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:09.736185 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:09.736149 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:10.040571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.040493 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j"] Oct 30 04:57:10.046158 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.046129 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.093437 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.089249 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fd27a9-openshift-default-dockercfg-zvwm5\"" Oct 30 04:57:10.140526 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.120136 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:10.140798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.121276 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-certs\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.140915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.140845 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nt9w\" (UniqueName: \"kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-kube-api-access-6nt9w\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.140915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.140890 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0fac68-9436-4c71-a3be-342e6054903e-istio-podinfo\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.140997 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.140956 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-socket\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.141042 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.141032 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-data\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.141095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.141080 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-credential-socket\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.141144 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.141130 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0fac68-9436-4c71-a3be-342e6054903e-istiod-ca-cert\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.141211 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.141195 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-istio-token\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.141354 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.141340 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-envoy\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.149303 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.149267 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j"] Oct 30 04:57:10.243116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243079 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-certs\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.243116 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243119 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nt9w\" (UniqueName: \"kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-kube-api-access-6nt9w\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.243374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243135 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0fac68-9436-4c71-a3be-342e6054903e-istio-podinfo\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.243374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243167 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-socket\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.243374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243199 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-data\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.243374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243225 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-credential-socket\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.243374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243255 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0fac68-9436-4c71-a3be-342e6054903e-istiod-ca-cert\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.243374 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243309 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-istio-token\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.243761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243434 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-envoy\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.243761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243614 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-certs\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.243761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243643 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-socket\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.243761 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243695 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-data\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.243994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.243786 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-credential-socket\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.244295 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.244275 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0fac68-9436-4c71-a3be-342e6054903e-istiod-ca-cert\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.245660 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.245633 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-envoy\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.245911 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.245892 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0fac68-9436-4c71-a3be-342e6054903e-istio-podinfo\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.294066 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.293970 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-istio-token\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.305724 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.305659 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nt9w\" (UniqueName: \"kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-kube-api-access-6nt9w\") pod \"e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.475449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.465724 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:10.539941 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.531896 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:10.762965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:10.762928 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:11.151469 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:11.151197 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j"] Oct 30 04:57:11.187736 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:57:11.187702 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0fac68_9436_4c71_a3be_342e6054903e.slice/crio-39ab53aa9fe9324734407dd500b538a9b7b8c5aa42b3a0834c5ab573bf2176b1 WatchSource:0}: Error finding container 39ab53aa9fe9324734407dd500b538a9b7b8c5aa42b3a0834c5ab573bf2176b1: Status 404 returned error can't find the container with id 39ab53aa9fe9324734407dd500b538a9b7b8c5aa42b3a0834c5ab573bf2176b1 Oct 30 04:57:11.193245 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:11.193187 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:57:11.193372 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:11.193293 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:57:11.193372 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:11.193342 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:57:11.225629 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:57:11.225593 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:57:11.832640 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:11.832608 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" event={"ID":"5f0fac68-9436-4c71-a3be-342e6054903e","Type":"ContainerStarted","Data":"f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e"} Oct 30 04:57:11.833056 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:11.832663 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" event={"ID":"5f0fac68-9436-4c71-a3be-342e6054903e","Type":"ContainerStarted","Data":"39ab53aa9fe9324734407dd500b538a9b7b8c5aa42b3a0834c5ab573bf2176b1"} Oct 30 04:57:11.906442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:11.904327 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" podStartSLOduration=1.9043063679999999 podStartE2EDuration="1.904306368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:57:11.896090886 +0000 UTC m=+3555.181965687" watchObservedRunningTime="2026-10-30 04:57:11.904306368 +0000 UTC m=+3555.190181143" Oct 30 04:57:12.052868 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:12.052826 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:12.480449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:12.480340 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:12.490808 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:12.490449 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:12.847600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:12.847522 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:12.847600 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:12.847588 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:57:13.909454 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:13.909392 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:15.528169 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:15.528019 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:16.092868 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:16.092835 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:16.579396 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:16.579306 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:16.802638 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:16.802603 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:18.087172 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:18.087123 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:18.334673 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:18.334641 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:18.549133 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:18.549095 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:19.978998 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:19.978950 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:20.136494 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:20.136460 4793 generic.go:356] "Generic (PLEG): container finished" podID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerID="7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698" exitCode=0 Oct 30 04:57:20.136665 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:20.136543 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" event={"ID":"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28","Type":"ContainerDied","Data":"7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698"} Oct 30 04:57:20.137016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:20.137000 4793 scope.go:122] "RemoveContainer" containerID="7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698" Oct 30 04:57:20.139012 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:20.138981 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:57:20.139095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:20.139057 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:57:20.139095 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:20.139090 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:57:21.146775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:21.146737 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" event={"ID":"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28","Type":"ContainerStarted","Data":"9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0"} Oct 30 04:57:21.667447 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:21.659890 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:21.945959 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:21.945927 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:57:21.947033 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:21.947003 4793 patch_prober.go:29] interesting pod/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 04:57:21.947166 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:21.947054 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 04:57:22.452842 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:22.452796 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:22.684474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:22.680366 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:22.945861 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:22.945826 4793 patch_prober.go:29] interesting pod/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 04:57:22.946044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:22.945878 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 04:57:23.332502 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:23.329210 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:23.946221 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:23.946182 4793 patch_prober.go:29] interesting pod/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 04:57:23.946746 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:23.946232 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 04:57:24.945846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:24.945789 4793 patch_prober.go:29] interesting pod/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 04:57:24.945846 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:24.945841 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 04:57:25.000442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:24.996388 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:25.377496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:25.376496 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:57:25.945698 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:25.945654 4793 patch_prober.go:29] interesting pod/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 04:57:25.945882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:25.945716 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 04:57:26.587043 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:26.587009 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:26.950652 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:26.950626 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:57:26.951882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:26.951860 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 04:57:28.335019 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:28.334982 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:30.004399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:30.004358 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:31.728776 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:31.728740 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:33.417586 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:33.417529 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:35.114704 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:35.114665 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:36.747857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:36.747816 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:38.441094 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:38.441056 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:40.213188 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:40.213141 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:41.220320 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:57:41.220288 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:57:41.955068 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:41.955028 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:43.629038 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:43.629004 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:45.390888 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:45.390851 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:46.977439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:46.977388 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:48.731338 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:48.731302 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:50.496443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:50.489679 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:52.224676 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:52.224628 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:54.113937 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:54.113893 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:56.009399 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:56.009344 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:57.323190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:57.323154 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:57.702857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:57.702821 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:57:59.405162 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:57:59.405125 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:01.062628 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:01.062593 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:01.278722 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:01.278684 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:02.817958 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:02.817919 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:04.567065 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:04.567027 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:06.440058 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:06.440027 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:07.987519 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:07.987481 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw"] Oct 30 04:58:07.993561 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:07.993529 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.009130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.009104 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw"] Oct 30 04:58:08.014915 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.014890 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fd27a9-serving-cert\"" Oct 30 04:58:08.167716 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.167680 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzbhv\" (UniqueName: \"kubernetes.io/projected/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-kube-api-access-jzbhv\") pod \"maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.167940 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.167913 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.168032 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.168013 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-metrics-certs\") pod \"maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.168239 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.168215 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-maas-api-tls\") pod \"maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.182146 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.182115 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:08.274232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.273730 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.274232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.273821 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-metrics-certs\") pod \"maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.274232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.274043 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-maas-api-tls\") pod \"maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.274232 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.274100 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzbhv\" (UniqueName: \"kubernetes.io/projected/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-kube-api-access-jzbhv\") pod \"maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.276769 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.275255 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.279756 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.279730 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-metrics-certs\") pod \"maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.279928 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.279816 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-maas-api-tls\") pod \"maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.298322 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.298295 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzbhv\" (UniqueName: \"kubernetes.io/projected/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-kube-api-access-jzbhv\") pod \"maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.311114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.311083 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:08.400073 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.399943 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:08.694593 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:58:08.694565 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod893d53f5_5c72_476e_a1d8_f2ed6ba5b9cf.slice/crio-32b28b087aace2a83cc205925bee0edcba5fcb5bbf08356e5bed324dfeda3358 WatchSource:0}: Error finding container 32b28b087aace2a83cc205925bee0edcba5fcb5bbf08356e5bed324dfeda3358: Status 404 returned error can't find the container with id 32b28b087aace2a83cc205925bee0edcba5fcb5bbf08356e5bed324dfeda3358 Oct 30 04:58:08.696307 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:08.696279 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw"] Oct 30 04:58:09.645642 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:09.645606 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" event={"ID":"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf","Type":"ContainerStarted","Data":"e2d00a3b31c13b6a6a10e19e9595b96eb7297ebda51c2796c3895b722c27e2ef"} Oct 30 04:58:09.646044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:09.645669 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" event={"ID":"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf","Type":"ContainerStarted","Data":"32b28b087aace2a83cc205925bee0edcba5fcb5bbf08356e5bed324dfeda3358"} Oct 30 04:58:09.646044 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:09.645707 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:09.690719 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:09.690666 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" podStartSLOduration=2.291266385 podStartE2EDuration="2.690653612s" totalImagesPullingTime="399.387227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:58:09.68625299 +0000 UTC m=+3612.972127764" watchObservedRunningTime="2026-10-30 04:58:09.690653612 +0000 UTC m=+3612.976528387" Oct 30 04:58:09.989571 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:09.989535 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:10.229025 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:10.228989 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:10.872218 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:10.872185 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn"] Oct 30 04:58:10.872695 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:10.872435 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" containerID="cri-o://95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9" gracePeriod=30 Oct 30 04:58:11.224096 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:58:11.224060 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:58:11.833243 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:11.833202 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:13.526781 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:13.526746 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:15.225621 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:15.225558 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:15.660729 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:15.660657 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 04:58:16.172506 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.172483 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:58:16.273275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.272730 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ec548c5-3019-48bc-9f48-e5894fb16bae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec548c5-3019-48bc-9f48-e5894fb16bae-istiod-ca-cert\") pod \"3ec548c5-3019-48bc-9f48-e5894fb16bae\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " Oct 30 04:58:16.273275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.272810 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-socket\") pod \"3ec548c5-3019-48bc-9f48-e5894fb16bae\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " Oct 30 04:58:16.274175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.273274 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-podinfo\") pod \"3ec548c5-3019-48bc-9f48-e5894fb16bae\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " Oct 30 04:58:16.274175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.273369 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-token\" (UniqueName: \"kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-token\") pod \"3ec548c5-3019-48bc-9f48-e5894fb16bae\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " Oct 30 04:58:16.274175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.273475 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-certs\") pod \"3ec548c5-3019-48bc-9f48-e5894fb16bae\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " Oct 30 04:58:16.274175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.273619 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-credential-socket\") pod \"3ec548c5-3019-48bc-9f48-e5894fb16bae\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " Oct 30 04:58:16.274175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.273727 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-data\") pod \"3ec548c5-3019-48bc-9f48-e5894fb16bae\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " Oct 30 04:58:16.274175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.273812 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-envoy\") pod \"3ec548c5-3019-48bc-9f48-e5894fb16bae\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " Oct 30 04:58:16.274175 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.273849 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-kube-api-access-k8qg6\" (UniqueName: \"kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-kube-api-access-k8qg6\") pod \"3ec548c5-3019-48bc-9f48-e5894fb16bae\" (UID: \"3ec548c5-3019-48bc-9f48-e5894fb16bae\") " Oct 30 04:58:16.275050 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.274792 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-certs" pod "3ec548c5-3019-48bc-9f48-e5894fb16bae" (UID: "3ec548c5-3019-48bc-9f48-e5894fb16bae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:58:16.275050 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.274839 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-credential-socket" pod "3ec548c5-3019-48bc-9f48-e5894fb16bae" (UID: "3ec548c5-3019-48bc-9f48-e5894fb16bae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:58:16.275050 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.274916 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-socket" pod "3ec548c5-3019-48bc-9f48-e5894fb16bae" (UID: "3ec548c5-3019-48bc-9f48-e5894fb16bae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:58:16.275282 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.275109 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec548c5-3019-48bc-9f48-e5894fb16bae-istiod-ca-cert" pod "3ec548c5-3019-48bc-9f48-e5894fb16bae" (UID: "3ec548c5-3019-48bc-9f48-e5894fb16bae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 04:58:16.275757 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.275694 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-data" pod "3ec548c5-3019-48bc-9f48-e5894fb16bae" (UID: "3ec548c5-3019-48bc-9f48-e5894fb16bae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:58:16.278844 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.278822 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-envoy" pod "3ec548c5-3019-48bc-9f48-e5894fb16bae" (UID: "3ec548c5-3019-48bc-9f48-e5894fb16bae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 04:58:16.279114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.279094 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-podinfo" pod "3ec548c5-3019-48bc-9f48-e5894fb16bae" (UID: "3ec548c5-3019-48bc-9f48-e5894fb16bae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 04:58:16.279329 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.279310 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-token" pod "3ec548c5-3019-48bc-9f48-e5894fb16bae" (UID: "3ec548c5-3019-48bc-9f48-e5894fb16bae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:58:16.283891 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.283870 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-kube-api-access-k8qg6" pod "3ec548c5-3019-48bc-9f48-e5894fb16bae" (UID: "3ec548c5-3019-48bc-9f48-e5894fb16bae"). InnerVolumeSpecName "kube-api-access-k8qg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 04:58:16.397767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.397394 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:58:16.397767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.397446 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:58:16.397767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.397460 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8qg6\" (UniqueName: \"kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-kube-api-access-k8qg6\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:58:16.397767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.397481 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ec548c5-3019-48bc-9f48-e5894fb16bae-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:58:16.397767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.397495 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:58:16.397767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.397507 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:58:16.397767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.397520 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ec548c5-3019-48bc-9f48-e5894fb16bae-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:58:16.397767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.397541 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:58:16.397767 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.397553 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ec548c5-3019-48bc-9f48-e5894fb16bae-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 04:58:16.702629 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.702591 4793 generic.go:356] "Generic (PLEG): container finished" podID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerID="95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9" exitCode=0 Oct 30 04:58:16.702828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.702656 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" Oct 30 04:58:16.702828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.702674 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" event={"ID":"3ec548c5-3019-48bc-9f48-e5894fb16bae","Type":"ContainerDied","Data":"95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9"} Oct 30 04:58:16.702828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.702757 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn" event={"ID":"3ec548c5-3019-48bc-9f48-e5894fb16bae","Type":"ContainerDied","Data":"d7fa63409c3e58844a2ab6d827c8edaae4bda9fc52b42275d25a66bd92626241"} Oct 30 04:58:16.702828 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.702782 4793 scope.go:122] "RemoveContainer" containerID="95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9" Oct 30 04:58:16.720785 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.720763 4793 scope.go:122] "RemoveContainer" containerID="95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9" Oct 30 04:58:16.721374 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:58:16.721351 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9\": container with ID starting with 95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9 not found: ID does not exist" containerID="95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9" Oct 30 04:58:16.721525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.721384 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9"} err="failed to get container status \"95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9\": rpc error: code = NotFound desc = could not find container \"95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9\": container with ID starting with 95b171e9e74860837e2cef90f13cabc69628f98fe5315a80d2e527a6539edda9 not found: ID does not exist" Oct 30 04:58:16.761658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.761628 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn"] Oct 30 04:58:16.771607 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.771574 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-79qpn"] Oct 30 04:58:16.849876 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.849845 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" path="/var/lib/kubelet/pods/3ec548c5-3019-48bc-9f48-e5894fb16bae/volumes" Oct 30 04:58:16.978798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:16.978700 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:18.731611 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:18.731576 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:19.064388 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:19.064298 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:19.598996 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:19.598954 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:20.178444 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:20.175202 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:20.413525 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:20.412379 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:20.721776 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:20.721740 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:22.395106 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:22.395063 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:24.245091 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:24.245057 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:24.466685 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:24.466642 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:26.028405 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:26.028366 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:26.244907 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:26.244868 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:28.023147 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:28.023112 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:29.709275 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:29.709229 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:31.478442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:31.463256 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:33.344441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:33.344311 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:35.130120 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:35.130082 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:36.814908 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:36.814871 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:38.437450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:38.437392 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:40.117970 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:40.117865 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:41.224265 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:58:41.224226 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:58:41.731964 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:41.731925 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:42.843775 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:42.843691 4793 patch_prober.go:29] interesting pod/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 04:58:42.844180 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:42.843773 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 04:58:43.000392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.000361 4793 generic.go:356] "Generic (PLEG): container finished" podID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerID="70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7" exitCode=0 Oct 30 04:58:43.000613 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.000467 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" event={"ID":"65430b4d-69e5-4e30-8a47-9c1e9003759c","Type":"ContainerDied","Data":"70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7"} Oct 30 04:58:43.000985 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.000965 4793 scope.go:122] "RemoveContainer" containerID="70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7" Oct 30 04:58:43.002999 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.002978 4793 generic.go:356] "Generic (PLEG): container finished" podID="5f0fac68-9436-4c71-a3be-342e6054903e" containerID="f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e" exitCode=0 Oct 30 04:58:43.003126 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.003054 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" event={"ID":"5f0fac68-9436-4c71-a3be-342e6054903e","Type":"ContainerDied","Data":"f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e"} Oct 30 04:58:43.003225 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.003193 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:58:43.003357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.003281 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:58:43.003357 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.003330 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:58:43.003535 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.003392 4793 scope.go:122] "RemoveContainer" containerID="f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e" Oct 30 04:58:43.005324 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.005296 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:58:43.005397 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.005361 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:58:43.005458 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.005397 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 04:58:43.532041 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:43.532005 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:44.015152 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:44.015110 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" event={"ID":"65430b4d-69e5-4e30-8a47-9c1e9003759c","Type":"ContainerStarted","Data":"6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41"} Oct 30 04:58:44.021585 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:44.021373 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" event={"ID":"5f0fac68-9436-4c71-a3be-342e6054903e","Type":"ContainerStarted","Data":"5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928"} Oct 30 04:58:44.479449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:44.478872 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:58:44.479449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:44.479010 4793 patch_prober.go:29] interesting pod/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 04:58:44.479449 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:44.479077 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 04:58:44.670859 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:44.670823 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:58:44.672468 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:44.672140 4793 patch_prober.go:29] interesting pod/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 04:58:44.672695 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:44.672672 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 04:58:45.174492 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:45.162026 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:45.480439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:45.475752 4793 patch_prober.go:29] interesting pod/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 04:58:45.480439 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:45.475812 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 04:58:45.690441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:45.673612 4793 patch_prober.go:29] interesting pod/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 04:58:45.690441 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:45.673671 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 04:58:46.474442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:46.473723 4793 patch_prober.go:29] interesting pod/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 04:58:46.474442 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:46.473784 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 04:58:46.675450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:46.674238 4793 patch_prober.go:29] interesting pod/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 04:58:46.675450 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:46.674310 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 04:58:46.823814 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:46.823722 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:47.477855 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:47.477811 4793 patch_prober.go:29] interesting pod/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 04:58:47.478655 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:47.478623 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 04:58:47.675890 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:47.675850 4793 patch_prober.go:29] interesting pod/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 04:58:47.676181 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:47.676152 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 04:58:48.478821 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:48.478786 4793 patch_prober.go:29] interesting pod/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 04:58:48.479481 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:48.479447 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 04:58:48.505965 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:48.505881 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:48.676228 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:48.676190 4793 patch_prober.go:29] interesting pod/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 04:58:48.680601 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:48.680564 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 04:58:49.484443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:49.483405 4793 patch_prober.go:29] interesting pod/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" start-of-body= Oct 30 04:58:49.484443 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:49.483490 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.231:15021/healthz/ready\": dial tcp 10.133.0.231:15021: connect: connection refused" Oct 30 04:58:49.669344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:49.668886 4793 patch_prober.go:29] interesting pod/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 04:58:49.669344 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:49.668949 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 04:58:50.124726 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:50.124680 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:50.479024 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:50.478990 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:58:50.479572 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:50.479543 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:58:50.480326 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:50.480304 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 04:58:50.674967 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:50.674935 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:58:50.675407 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:50.675392 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:58:50.676336 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:50.676188 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 04:58:51.795619 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:51.795519 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:53.468100 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:53.468057 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:55.234521 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:55.234481 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:57.031798 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:57.031767 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:57.445884 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:57.445852 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:58:58.710512 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:58:58.710474 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:00.587751 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:00.587704 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:00.912433 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:00.912326 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:01.436392 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:01.436359 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:01.922671 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:01.922584 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:02.141472 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:02.141437 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:02.357393 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:02.357355 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:04.032340 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:04.032292 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:05.953237 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:05.953193 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:06.171016 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:06.170979 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:07.025872 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.025826 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:07.723919 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.723881 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:07.775664 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.775630 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9"] Oct 30 04:59:07.776461 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.776445 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" Oct 30 04:59:07.776511 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.776462 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" Oct 30 04:59:07.776573 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.776564 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ec548c5-3019-48bc-9f48-e5894fb16bae" containerName="istio-proxy" Oct 30 04:59:07.780670 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.780641 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" Oct 30 04:59:07.808860 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.808832 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9"] Oct 30 04:59:07.848217 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.848183 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-26b72f\"" Oct 30 04:59:07.848634 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.848614 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-26b72f-dockercfg-st896\"" Oct 30 04:59:07.900919 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.900889 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sntlg\" (UniqueName: \"kubernetes.io/projected/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-kube-api-access-sntlg\") pod \"payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9\" (UID: \"357e05d0-d457-4cb2-8d86-4a7990e6d6eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" Oct 30 04:59:07.901075 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.900973 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9\" (UID: \"357e05d0-d457-4cb2-8d86-4a7990e6d6eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" Oct 30 04:59:07.980496 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:07.980274 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:08.023446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.021381 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sntlg\" (UniqueName: \"kubernetes.io/projected/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-kube-api-access-sntlg\") pod \"payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9\" (UID: \"357e05d0-d457-4cb2-8d86-4a7990e6d6eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" Oct 30 04:59:08.023446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.021493 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9\" (UID: \"357e05d0-d457-4cb2-8d86-4a7990e6d6eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" Oct 30 04:59:08.023446 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.022784 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9\" (UID: \"357e05d0-d457-4cb2-8d86-4a7990e6d6eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" Oct 30 04:59:08.173438 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.165235 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sntlg\" (UniqueName: \"kubernetes.io/projected/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-kube-api-access-sntlg\") pod \"payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9\" (UID: \"357e05d0-d457-4cb2-8d86-4a7990e6d6eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" Oct 30 04:59:08.198561 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.198527 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:08.403132 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.403057 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" Oct 30 04:59:08.792085 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.792054 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9"] Oct 30 04:59:08.792359 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:59:08.792338 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357e05d0_d457_4cb2_8d86_4a7990e6d6eb.slice/crio-e49b0aeae3694b9ea797016e7c0cb314642985fc5d50fd1fedb9e6ea75a95594 WatchSource:0}: Error finding container e49b0aeae3694b9ea797016e7c0cb314642985fc5d50fd1fedb9e6ea75a95594: Status 404 returned error can't find the container with id e49b0aeae3694b9ea797016e7c0cb314642985fc5d50fd1fedb9e6ea75a95594 Oct 30 04:59:08.932200 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.932169 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9"] Oct 30 04:59:08.937065 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.937043 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:08.964677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.952978 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-449d83\"/\"openshift-service-ca.crt\"" Oct 30 04:59:08.964677 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.958857 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9"] Oct 30 04:59:08.970694 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.970663 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-449d83\"/\"test-model-449d83-kserve-self-signed-certs\"" Oct 30 04:59:08.970857 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.970741 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-449d83\"/\"kube-root-ca.crt\"" Oct 30 04:59:08.971391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:08.971368 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-449d83\"/\"default-dockercfg-tzwvm\"" Oct 30 04:59:09.043034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.042943 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-home\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.043186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.043093 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-model-cache\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.043186 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.043174 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-dshm\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.043391 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.043364 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5x5r\" (UniqueName: \"kubernetes.io/projected/667e7bbb-b453-4597-ad84-f529a4f8d8b1-kube-api-access-b5x5r\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.043630 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.043603 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tls-certs\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.043776 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.043756 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tmp-dir\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.109252 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.109217 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:09.145190 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.145155 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tls-certs\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.145337 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.145252 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tmp-dir\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.145337 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.145276 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-home\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.145474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.145334 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-model-cache\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.145474 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.145399 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-dshm\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.145590 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.145524 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5x5r\" (UniqueName: \"kubernetes.io/projected/667e7bbb-b453-4597-ad84-f529a4f8d8b1-kube-api-access-b5x5r\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.145877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.145851 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-home\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.145877 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.145863 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-model-cache\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.146115 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.146088 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tmp-dir\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.147783 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.147755 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-dshm\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.148297 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.148276 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tls-certs\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.216603 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.203395 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5x5r\" (UniqueName: \"kubernetes.io/projected/667e7bbb-b453-4597-ad84-f529a4f8d8b1-kube-api-access-b5x5r\") pod \"test-model-449d83-kserve-6f468d4bc-mdzc9\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.254938 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.254673 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:09.345654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.345537 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" event={"ID":"357e05d0-d457-4cb2-8d86-4a7990e6d6eb","Type":"ContainerStarted","Data":"fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96"} Oct 30 04:59:09.345654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.345618 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" Oct 30 04:59:09.345654 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.345643 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" event={"ID":"357e05d0-d457-4cb2-8d86-4a7990e6d6eb","Type":"ContainerStarted","Data":"e49b0aeae3694b9ea797016e7c0cb314642985fc5d50fd1fedb9e6ea75a95594"} Oct 30 04:59:09.392448 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.392380 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" podStartSLOduration=2.39236098 podStartE2EDuration="2.39236098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:09.387153881 +0000 UTC m=+3672.673028658" watchObservedRunningTime="2026-10-30 04:59:09.39236098 +0000 UTC m=+3672.678235757" Oct 30 04:59:09.619281 ip-10-0-138-177 kubenswrapper[4793]: W1030 04:59:09.619250 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod667e7bbb_b453_4597_ad84_f529a4f8d8b1.slice/crio-0158d17b3ea926687d5aaede5933a15dc5e13eede5d532aa38d5d3acf52f63f6 WatchSource:0}: Error finding container 0158d17b3ea926687d5aaede5933a15dc5e13eede5d532aa38d5d3acf52f63f6: Status 404 returned error can't find the container with id 0158d17b3ea926687d5aaede5933a15dc5e13eede5d532aa38d5d3acf52f63f6 Oct 30 04:59:09.621943 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.621915 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9"] Oct 30 04:59:09.730786 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:09.730751 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:10.355951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:10.355903 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" event={"ID":"667e7bbb-b453-4597-ad84-f529a4f8d8b1","Type":"ContainerStarted","Data":"8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c"} Oct 30 04:59:10.355951 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:10.355958 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" event={"ID":"667e7bbb-b453-4597-ad84-f529a4f8d8b1","Type":"ContainerStarted","Data":"0158d17b3ea926687d5aaede5933a15dc5e13eede5d532aa38d5d3acf52f63f6"} Oct 30 04:59:10.396674 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:10.396619 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" podStartSLOduration=2.396605544 podStartE2EDuration="2.396605544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:59:10.395671569 +0000 UTC m=+3673.681546343" watchObservedRunningTime="2026-10-30 04:59:10.396605544 +0000 UTC m=+3673.682480321" Oct 30 04:59:11.220720 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:59:11.220681 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:59:11.255568 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:11.255539 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:11.412563 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:11.412525 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:13.017114 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:13.017071 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:14.646159 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:14.646114 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:16.779731 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:16.779691 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:18.128130 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:18.128089 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:19.256031 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:19.256002 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:19.275316 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:19.275289 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:19.457863 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:19.457834 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 04:59:19.939840 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:19.939803 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:20.166317 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:20.166284 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:20.360192 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:20.360111 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" Oct 30 04:59:21.828263 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:21.828228 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:23.715945 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:23.715906 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:25.457610 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:25.457570 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:27.087435 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:27.087380 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:29.027069 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:29.027038 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:29.646362 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:29.646323 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:30.887994 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:30.887959 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:32.508475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:32.508378 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:34.230185 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:34.230144 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:35.906475 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:35.906436 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:37.529202 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:37.529167 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:39.268377 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:39.268316 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:40.989810 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:40.989773 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:41.221115 ip-10-0-138-177 kubenswrapper[4793]: E1030 04:59:41.221072 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 04:59:42.699077 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:42.699040 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:44.439960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:44.439930 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:46.241865 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:46.241833 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:47.869960 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:47.869920 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:49.643554 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:49.643512 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:51.346610 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:51.346571 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:53.036882 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:53.036845 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:54.896298 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:54.896265 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:56.619658 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:56.619613 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:57.574034 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:57.573989 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 04:59:58.425952 ip-10-0-138-177 kubenswrapper[4793]: I1030 04:59:58.425909 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:00.037734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.037702 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:00.250298 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.250246 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4"] Oct 30 05:00:00.264133 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.264103 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57"] Oct 30 05:00:00.264575 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.264465 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" Oct 30 05:00:00.271302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.271270 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb"] Oct 30 05:00:00.271628 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.271410 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" Oct 30 05:00:00.282111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.282083 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 05:00:00.285612 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.285590 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" Oct 30 05:00:00.289267 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.289207 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57"] Oct 30 05:00:00.298916 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.298889 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4"] Oct 30 05:00:00.308495 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.308399 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf"] Oct 30 05:00:00.314950 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.314619 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" Oct 30 05:00:00.315366 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.315342 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb"] Oct 30 05:00:00.320799 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.320774 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn"] Oct 30 05:00:00.348238 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.348208 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5"] Oct 30 05:00:00.349526 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.349504 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" Oct 30 05:00:00.353515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.353494 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn"] Oct 30 05:00:00.353634 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.353545 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b"] Oct 30 05:00:00.353835 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.353783 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" Oct 30 05:00:00.359660 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.359638 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" Oct 30 05:00:00.364838 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.364816 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn"] Oct 30 05:00:00.371681 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.371657 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" Oct 30 05:00:00.372588 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.372562 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf"] Oct 30 05:00:00.382413 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.382369 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b"] Oct 30 05:00:00.387062 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.387037 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5"] Oct 30 05:00:00.400888 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.400865 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz"] Oct 30 05:00:00.412659 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.412638 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42"] Oct 30 05:00:00.414177 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.412865 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" Oct 30 05:00:00.420242 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.420222 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn"] Oct 30 05:00:00.420604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.420448 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" Oct 30 05:00:00.421351 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.421327 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz"] Oct 30 05:00:00.433329 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.433301 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42"] Oct 30 05:00:00.441476 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.440938 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb\" (UID: \"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" Oct 30 05:00:00.441476 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.441046 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zz7c\" (UniqueName: \"kubernetes.io/projected/1f66a424-269e-475e-bf86-3e40027c9934-kube-api-access-9zz7c\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57\" (UID: \"1f66a424-269e-475e-bf86-3e40027c9934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" Oct 30 05:00:00.441476 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.441102 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjfls\" (UniqueName: \"kubernetes.io/projected/d0225097-2d12-43e4-89b8-ea6b75c9ea54-kube-api-access-pjfls\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf\" (UID: \"d0225097-2d12-43e4-89b8-ea6b75c9ea54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" Oct 30 05:00:00.441476 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.441123 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b116c3-eb39-4ffb-b765-666e2a9a9966-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5\" (UID: \"f0b116c3-eb39-4ffb-b765-666e2a9a9966\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" Oct 30 05:00:00.441476 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.441185 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a17c173-623d-4564-9f4c-d590922b6520-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn\" (UID: \"1a17c173-623d-4564-9f4c-d590922b6520\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" Oct 30 05:00:00.441476 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.441242 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-622hh\" (UniqueName: \"kubernetes.io/projected/43b309b4-549a-48c2-9260-fc1ae51dd264-kube-api-access-622hh\") pod \"maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4\" (UID: \"43b309b4-549a-48c2-9260-fc1ae51dd264\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" Oct 30 05:00:00.441476 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.441278 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f66a424-269e-475e-bf86-3e40027c9934-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57\" (UID: \"1f66a424-269e-475e-bf86-3e40027c9934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" Oct 30 05:00:00.441476 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.441310 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpkcs\" (UniqueName: \"kubernetes.io/projected/f0b116c3-eb39-4ffb-b765-666e2a9a9966-kube-api-access-zpkcs\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5\" (UID: \"f0b116c3-eb39-4ffb-b765-666e2a9a9966\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" Oct 30 05:00:00.441476 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.441360 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0225097-2d12-43e4-89b8-ea6b75c9ea54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf\" (UID: \"d0225097-2d12-43e4-89b8-ea6b75c9ea54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" Oct 30 05:00:00.441476 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.441393 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b309b4-549a-48c2-9260-fc1ae51dd264-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4\" (UID: \"43b309b4-549a-48c2-9260-fc1ae51dd264\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" Oct 30 05:00:00.442120 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.441539 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txffq\" (UniqueName: \"kubernetes.io/projected/1a17c173-623d-4564-9f4c-d590922b6520-kube-api-access-txffq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn\" (UID: \"1a17c173-623d-4564-9f4c-d590922b6520\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" Oct 30 05:00:00.442120 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.441717 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zm8l\" (UniqueName: \"kubernetes.io/projected/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-kube-api-access-8zm8l\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb\" (UID: \"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" Oct 30 05:00:00.542869 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.542797 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c38420-6bf8-4305-bbbf-6656194e1003-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn\" (UID: \"54c38420-6bf8-4305-bbbf-6656194e1003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" Oct 30 05:00:00.542869 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.542838 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zm8l\" (UniqueName: \"kubernetes.io/projected/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-kube-api-access-8zm8l\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb\" (UID: \"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" Oct 30 05:00:00.543102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.542958 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb\" (UID: \"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" Oct 30 05:00:00.543102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543041 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zz7c\" (UniqueName: \"kubernetes.io/projected/1f66a424-269e-475e-bf86-3e40027c9934-kube-api-access-9zz7c\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57\" (UID: \"1f66a424-269e-475e-bf86-3e40027c9934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" Oct 30 05:00:00.543102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543082 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjfls\" (UniqueName: \"kubernetes.io/projected/d0225097-2d12-43e4-89b8-ea6b75c9ea54-kube-api-access-pjfls\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf\" (UID: \"d0225097-2d12-43e4-89b8-ea6b75c9ea54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" Oct 30 05:00:00.543102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543103 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b116c3-eb39-4ffb-b765-666e2a9a9966-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5\" (UID: \"f0b116c3-eb39-4ffb-b765-666e2a9a9966\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" Oct 30 05:00:00.543340 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543210 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3090898b-f127-44a1-85c3-ca256c7cb642-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b\" (UID: \"3090898b-f127-44a1-85c3-ca256c7cb642\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" Oct 30 05:00:00.543340 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543263 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c94wx\" (UniqueName: \"kubernetes.io/projected/3090898b-f127-44a1-85c3-ca256c7cb642-kube-api-access-c94wx\") pod \"maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b\" (UID: \"3090898b-f127-44a1-85c3-ca256c7cb642\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" Oct 30 05:00:00.543512 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543341 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a17c173-623d-4564-9f4c-d590922b6520-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn\" (UID: \"1a17c173-623d-4564-9f4c-d590922b6520\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" Oct 30 05:00:00.543512 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543377 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrsh8\" (UniqueName: \"kubernetes.io/projected/44d00032-6fea-4d6d-93d8-e481e477cc96-kube-api-access-mrsh8\") pod \"maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42\" (UID: \"44d00032-6fea-4d6d-93d8-e481e477cc96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" Oct 30 05:00:00.543512 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543468 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-622hh\" (UniqueName: \"kubernetes.io/projected/43b309b4-549a-48c2-9260-fc1ae51dd264-kube-api-access-622hh\") pod \"maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4\" (UID: \"43b309b4-549a-48c2-9260-fc1ae51dd264\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" Oct 30 05:00:00.543704 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543529 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f66a424-269e-475e-bf86-3e40027c9934-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57\" (UID: \"1f66a424-269e-475e-bf86-3e40027c9934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" Oct 30 05:00:00.543704 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543667 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d00032-6fea-4d6d-93d8-e481e477cc96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42\" (UID: \"44d00032-6fea-4d6d-93d8-e481e477cc96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" Oct 30 05:00:00.543817 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543729 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpkcs\" (UniqueName: \"kubernetes.io/projected/f0b116c3-eb39-4ffb-b765-666e2a9a9966-kube-api-access-zpkcs\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5\" (UID: \"f0b116c3-eb39-4ffb-b765-666e2a9a9966\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" Oct 30 05:00:00.543875 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543821 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b116c3-eb39-4ffb-b765-666e2a9a9966-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5\" (UID: \"f0b116c3-eb39-4ffb-b765-666e2a9a9966\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" Oct 30 05:00:00.543875 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543828 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb\" (UID: \"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" Oct 30 05:00:00.543875 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543842 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxtj5\" (UniqueName: \"kubernetes.io/projected/291b86ef-89c7-4b35-b649-e90aa681670e-kube-api-access-lxtj5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz\" (UID: \"291b86ef-89c7-4b35-b649-e90aa681670e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" Oct 30 05:00:00.543984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543896 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0225097-2d12-43e4-89b8-ea6b75c9ea54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf\" (UID: \"d0225097-2d12-43e4-89b8-ea6b75c9ea54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" Oct 30 05:00:00.543984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543924 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2ngv\" (UniqueName: \"kubernetes.io/projected/54c38420-6bf8-4305-bbbf-6656194e1003-kube-api-access-k2ngv\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn\" (UID: \"54c38420-6bf8-4305-bbbf-6656194e1003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" Oct 30 05:00:00.543984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.543962 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b309b4-549a-48c2-9260-fc1ae51dd264-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4\" (UID: \"43b309b4-549a-48c2-9260-fc1ae51dd264\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" Oct 30 05:00:00.544094 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.544016 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291b86ef-89c7-4b35-b649-e90aa681670e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz\" (UID: \"291b86ef-89c7-4b35-b649-e90aa681670e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" Oct 30 05:00:00.544134 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.544093 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txffq\" (UniqueName: \"kubernetes.io/projected/1a17c173-623d-4564-9f4c-d590922b6520-kube-api-access-txffq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn\" (UID: \"1a17c173-623d-4564-9f4c-d590922b6520\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" Oct 30 05:00:00.544134 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.544107 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a17c173-623d-4564-9f4c-d590922b6520-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn\" (UID: \"1a17c173-623d-4564-9f4c-d590922b6520\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" Oct 30 05:00:00.544269 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.544247 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f66a424-269e-475e-bf86-3e40027c9934-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57\" (UID: \"1f66a424-269e-475e-bf86-3e40027c9934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" Oct 30 05:00:00.544796 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.544774 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0225097-2d12-43e4-89b8-ea6b75c9ea54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf\" (UID: \"d0225097-2d12-43e4-89b8-ea6b75c9ea54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" Oct 30 05:00:00.544863 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.544843 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b309b4-549a-48c2-9260-fc1ae51dd264-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4\" (UID: \"43b309b4-549a-48c2-9260-fc1ae51dd264\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" Oct 30 05:00:00.568306 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.568277 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zz7c\" (UniqueName: \"kubernetes.io/projected/1f66a424-269e-475e-bf86-3e40027c9934-kube-api-access-9zz7c\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57\" (UID: \"1f66a424-269e-475e-bf86-3e40027c9934\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" Oct 30 05:00:00.568463 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.568386 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjfls\" (UniqueName: \"kubernetes.io/projected/d0225097-2d12-43e4-89b8-ea6b75c9ea54-kube-api-access-pjfls\") pod \"maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf\" (UID: \"d0225097-2d12-43e4-89b8-ea6b75c9ea54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" Oct 30 05:00:00.568551 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.568525 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-622hh\" (UniqueName: \"kubernetes.io/projected/43b309b4-549a-48c2-9260-fc1ae51dd264-kube-api-access-622hh\") pod \"maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4\" (UID: \"43b309b4-549a-48c2-9260-fc1ae51dd264\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" Oct 30 05:00:00.569184 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.569161 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txffq\" (UniqueName: \"kubernetes.io/projected/1a17c173-623d-4564-9f4c-d590922b6520-kube-api-access-txffq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn\" (UID: \"1a17c173-623d-4564-9f4c-d590922b6520\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" Oct 30 05:00:00.573027 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.573003 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpkcs\" (UniqueName: \"kubernetes.io/projected/f0b116c3-eb39-4ffb-b765-666e2a9a9966-kube-api-access-zpkcs\") pod \"maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5\" (UID: \"f0b116c3-eb39-4ffb-b765-666e2a9a9966\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" Oct 30 05:00:00.574716 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.574696 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zm8l\" (UniqueName: \"kubernetes.io/projected/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-kube-api-access-8zm8l\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb\" (UID: \"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" Oct 30 05:00:00.630111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.630090 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" Oct 30 05:00:00.646021 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.645995 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3090898b-f127-44a1-85c3-ca256c7cb642-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b\" (UID: \"3090898b-f127-44a1-85c3-ca256c7cb642\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" Oct 30 05:00:00.646152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.646053 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c94wx\" (UniqueName: \"kubernetes.io/projected/3090898b-f127-44a1-85c3-ca256c7cb642-kube-api-access-c94wx\") pod \"maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b\" (UID: \"3090898b-f127-44a1-85c3-ca256c7cb642\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" Oct 30 05:00:00.646152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.646103 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrsh8\" (UniqueName: \"kubernetes.io/projected/44d00032-6fea-4d6d-93d8-e481e477cc96-kube-api-access-mrsh8\") pod \"maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42\" (UID: \"44d00032-6fea-4d6d-93d8-e481e477cc96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" Oct 30 05:00:00.646152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.646139 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" Oct 30 05:00:00.646376 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.646354 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d00032-6fea-4d6d-93d8-e481e477cc96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42\" (UID: \"44d00032-6fea-4d6d-93d8-e481e477cc96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" Oct 30 05:00:00.646467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.646450 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxtj5\" (UniqueName: \"kubernetes.io/projected/291b86ef-89c7-4b35-b649-e90aa681670e-kube-api-access-lxtj5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz\" (UID: \"291b86ef-89c7-4b35-b649-e90aa681670e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" Oct 30 05:00:00.646522 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.646492 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2ngv\" (UniqueName: \"kubernetes.io/projected/54c38420-6bf8-4305-bbbf-6656194e1003-kube-api-access-k2ngv\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn\" (UID: \"54c38420-6bf8-4305-bbbf-6656194e1003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" Oct 30 05:00:00.646560 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.646547 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291b86ef-89c7-4b35-b649-e90aa681670e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz\" (UID: \"291b86ef-89c7-4b35-b649-e90aa681670e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" Oct 30 05:00:00.646840 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.646661 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c38420-6bf8-4305-bbbf-6656194e1003-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn\" (UID: \"54c38420-6bf8-4305-bbbf-6656194e1003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" Oct 30 05:00:00.646840 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.646695 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3090898b-f127-44a1-85c3-ca256c7cb642-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b\" (UID: \"3090898b-f127-44a1-85c3-ca256c7cb642\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" Oct 30 05:00:00.647232 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.647190 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d00032-6fea-4d6d-93d8-e481e477cc96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42\" (UID: \"44d00032-6fea-4d6d-93d8-e481e477cc96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" Oct 30 05:00:00.647470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.647448 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c38420-6bf8-4305-bbbf-6656194e1003-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn\" (UID: \"54c38420-6bf8-4305-bbbf-6656194e1003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" Oct 30 05:00:00.647534 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.647487 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291b86ef-89c7-4b35-b649-e90aa681670e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz\" (UID: \"291b86ef-89c7-4b35-b649-e90aa681670e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" Oct 30 05:00:00.660677 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.660652 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" Oct 30 05:00:00.673699 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.673649 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c94wx\" (UniqueName: \"kubernetes.io/projected/3090898b-f127-44a1-85c3-ca256c7cb642-kube-api-access-c94wx\") pod \"maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b\" (UID: \"3090898b-f127-44a1-85c3-ca256c7cb642\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" Oct 30 05:00:00.684496 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.678514 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2ngv\" (UniqueName: \"kubernetes.io/projected/54c38420-6bf8-4305-bbbf-6656194e1003-kube-api-access-k2ngv\") pod \"maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn\" (UID: \"54c38420-6bf8-4305-bbbf-6656194e1003\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" Oct 30 05:00:00.684496 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.680207 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrsh8\" (UniqueName: \"kubernetes.io/projected/44d00032-6fea-4d6d-93d8-e481e477cc96-kube-api-access-mrsh8\") pod \"maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42\" (UID: \"44d00032-6fea-4d6d-93d8-e481e477cc96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" Oct 30 05:00:00.688526 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.688497 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxtj5\" (UniqueName: \"kubernetes.io/projected/291b86ef-89c7-4b35-b649-e90aa681670e-kube-api-access-lxtj5\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz\" (UID: \"291b86ef-89c7-4b35-b649-e90aa681670e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" Oct 30 05:00:00.690849 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.690816 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" Oct 30 05:00:00.722814 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.722595 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" Oct 30 05:00:00.733817 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.733463 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" Oct 30 05:00:00.739102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.737634 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" Oct 30 05:00:00.744844 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.744665 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" Oct 30 05:00:00.752087 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.751728 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" Oct 30 05:00:00.770783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.769623 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" Oct 30 05:00:00.980202 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:00.980146 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4"] Oct 30 05:00:01.367963 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:00:01.367933 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a17c173_623d_4564_9f4c_d590922b6520.slice/crio-6de7ca6eac409bb1942756e2acd98901e7ee68c0bc889c3a7e6d7c0d256dad07 WatchSource:0}: Error finding container 6de7ca6eac409bb1942756e2acd98901e7ee68c0bc889c3a7e6d7c0d256dad07: Status 404 returned error can't find the container with id 6de7ca6eac409bb1942756e2acd98901e7ee68c0bc889c3a7e6d7c0d256dad07 Oct 30 05:00:01.370077 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.370047 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn"] Oct 30 05:00:01.384826 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.384796 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz"] Oct 30 05:00:01.396885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.396861 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57"] Oct 30 05:00:01.402304 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.402279 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42"] Oct 30 05:00:01.409177 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.409152 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb"] Oct 30 05:00:01.416285 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:00:01.416257 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f66a424_269e_475e_bf86_3e40027c9934.slice/crio-951c5246548d786c6dc6cac6a9a3880a9e3ed40c74e9ed542051be0bcb8e12a0 WatchSource:0}: Error finding container 951c5246548d786c6dc6cac6a9a3880a9e3ed40c74e9ed542051be0bcb8e12a0: Status 404 returned error can't find the container with id 951c5246548d786c6dc6cac6a9a3880a9e3ed40c74e9ed542051be0bcb8e12a0 Oct 30 05:00:01.416714 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:00:01.416693 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d00032_6fea_4d6d_93d8_e481e477cc96.slice/crio-767cd014d6d13672ece3f7959572181688a1615c068b1c5352e4967087a86f81 WatchSource:0}: Error finding container 767cd014d6d13672ece3f7959572181688a1615c068b1c5352e4967087a86f81: Status 404 returned error can't find the container with id 767cd014d6d13672ece3f7959572181688a1615c068b1c5352e4967087a86f81 Oct 30 05:00:01.417031 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:00:01.417009 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf9c38c1_b1d7_493f_9a0d_9bba8651cbb8.slice/crio-ee263736cfe5f4064632bb6abf7b0cb7ce96e564835e664303abec2ba1899d3c WatchSource:0}: Error finding container ee263736cfe5f4064632bb6abf7b0cb7ce96e564835e664303abec2ba1899d3c: Status 404 returned error can't find the container with id ee263736cfe5f4064632bb6abf7b0cb7ce96e564835e664303abec2ba1899d3c Oct 30 05:00:01.461018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.460988 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf"] Oct 30 05:00:01.490518 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.490489 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b"] Oct 30 05:00:01.537841 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.537816 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5"] Oct 30 05:00:01.559498 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:00:01.559471 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0225097_2d12_43e4_89b8_ea6b75c9ea54.slice/crio-0e1c6df9316a922f3b82e7279562d7717638057b23c544041c0387a2557dbbf4 WatchSource:0}: Error finding container 0e1c6df9316a922f3b82e7279562d7717638057b23c544041c0387a2557dbbf4: Status 404 returned error can't find the container with id 0e1c6df9316a922f3b82e7279562d7717638057b23c544041c0387a2557dbbf4 Oct 30 05:00:01.578856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.578820 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn"] Oct 30 05:00:01.661176 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:00:01.660265 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3090898b_f127_44a1_85c3_ca256c7cb642.slice/crio-80323c6c4bd6f41e40934dabbdd3c740533c731ad6f6ae92c3ca37823f8de1a0 WatchSource:0}: Error finding container 80323c6c4bd6f41e40934dabbdd3c740533c731ad6f6ae92c3ca37823f8de1a0: Status 404 returned error can't find the container with id 80323c6c4bd6f41e40934dabbdd3c740533c731ad6f6ae92c3ca37823f8de1a0 Oct 30 05:00:01.661700 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:00:01.661599 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b116c3_eb39_4ffb_b765_666e2a9a9966.slice/crio-3c2624f28a230ce615fd36d923466bbe1e0b5127769d5e49462ccceccc74301c WatchSource:0}: Error finding container 3c2624f28a230ce615fd36d923466bbe1e0b5127769d5e49462ccceccc74301c: Status 404 returned error can't find the container with id 3c2624f28a230ce615fd36d923466bbe1e0b5127769d5e49462ccceccc74301c Oct 30 05:00:01.665861 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:00:01.665458 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c38420_6bf8_4305_bbbf_6656194e1003.slice/crio-804a4d2d1361a71c2cd332ac72616de96458017e16c0eb2ebb10d033ba8b06db WatchSource:0}: Error finding container 804a4d2d1361a71c2cd332ac72616de96458017e16c0eb2ebb10d033ba8b06db: Status 404 returned error can't find the container with id 804a4d2d1361a71c2cd332ac72616de96458017e16c0eb2ebb10d033ba8b06db Oct 30 05:00:01.945966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.945933 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" event={"ID":"1a17c173-623d-4564-9f4c-d590922b6520","Type":"ContainerStarted","Data":"aa136c158357a505b0e3847ea09cdec9c5e4606ba4810848ba4b0cb934c47fa6"} Oct 30 05:00:01.946107 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.945994 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" event={"ID":"1a17c173-623d-4564-9f4c-d590922b6520","Type":"ContainerStarted","Data":"6de7ca6eac409bb1942756e2acd98901e7ee68c0bc889c3a7e6d7c0d256dad07"} Oct 30 05:00:01.947485 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.947459 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" event={"ID":"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8","Type":"ContainerStarted","Data":"ee263736cfe5f4064632bb6abf7b0cb7ce96e564835e664303abec2ba1899d3c"} Oct 30 05:00:01.948851 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.948825 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" event={"ID":"d0225097-2d12-43e4-89b8-ea6b75c9ea54","Type":"ContainerStarted","Data":"0e1c6df9316a922f3b82e7279562d7717638057b23c544041c0387a2557dbbf4"} Oct 30 05:00:01.950455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.950413 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" event={"ID":"1f66a424-269e-475e-bf86-3e40027c9934","Type":"ContainerStarted","Data":"951c5246548d786c6dc6cac6a9a3880a9e3ed40c74e9ed542051be0bcb8e12a0"} Oct 30 05:00:01.952260 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:01.952190 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" event={"ID":"291b86ef-89c7-4b35-b649-e90aa681670e","Type":"ContainerStarted","Data":"5f7b87fe5c032c0891fb3a1d65ef7b6f4c0b8b5c56c9998ec503395f58c51a3f"} Oct 30 05:00:02.019303 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:02.019266 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" event={"ID":"44d00032-6fea-4d6d-93d8-e481e477cc96","Type":"ContainerStarted","Data":"767cd014d6d13672ece3f7959572181688a1615c068b1c5352e4967087a86f81"} Oct 30 05:00:02.020658 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:02.020277 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" podStartSLOduration=2.020259043 podStartE2EDuration="2.020259043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:01.992757212 +0000 UTC m=+3725.278631988" watchObservedRunningTime="2026-10-30 05:00:02.020259043 +0000 UTC m=+3725.306133819" Oct 30 05:00:02.021394 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:02.021134 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:02.021873 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:02.021661 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" event={"ID":"54c38420-6bf8-4305-bbbf-6656194e1003","Type":"ContainerStarted","Data":"804a4d2d1361a71c2cd332ac72616de96458017e16c0eb2ebb10d033ba8b06db"} Oct 30 05:00:02.023834 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:02.023810 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" event={"ID":"f0b116c3-eb39-4ffb-b765-666e2a9a9966","Type":"ContainerStarted","Data":"3c2624f28a230ce615fd36d923466bbe1e0b5127769d5e49462ccceccc74301c"} Oct 30 05:00:02.026498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:02.026472 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" event={"ID":"3090898b-f127-44a1-85c3-ca256c7cb642","Type":"ContainerStarted","Data":"80323c6c4bd6f41e40934dabbdd3c740533c731ad6f6ae92c3ca37823f8de1a0"} Oct 30 05:00:02.029268 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:02.029230 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" event={"ID":"43b309b4-549a-48c2-9260-fc1ae51dd264","Type":"ContainerStarted","Data":"30ef2873478e6fbb687cbd32684d476d0defc96bdedaf87cae889fa7eba6a9ed"} Oct 30 05:00:02.029370 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:02.029291 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" event={"ID":"43b309b4-549a-48c2-9260-fc1ae51dd264","Type":"ContainerStarted","Data":"28969d300e2391c9f90566897ad01e5ef282f499f1124fc13af47351dd810630"} Oct 30 05:00:02.076172 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:02.076120 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" podStartSLOduration=2.076107218 podStartE2EDuration="2.076107218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:02.07287889 +0000 UTC m=+3725.358753690" watchObservedRunningTime="2026-10-30 05:00:02.076107218 +0000 UTC m=+3725.361981993" Oct 30 05:00:03.040242 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.040213 4793 generic.go:356] "Generic (PLEG): container finished" podID="54c38420-6bf8-4305-bbbf-6656194e1003" containerID="fb9e815c70fd8ac2ff0e6efdc7e18310e6faa3dc85edfd5bc5d392fb23e6e344" exitCode=60 Oct 30 05:00:03.040687 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.040299 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" event={"ID":"54c38420-6bf8-4305-bbbf-6656194e1003","Type":"ContainerDied","Data":"fb9e815c70fd8ac2ff0e6efdc7e18310e6faa3dc85edfd5bc5d392fb23e6e344"} Oct 30 05:00:03.040687 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.040612 4793 scope.go:122] "RemoveContainer" containerID="fb9e815c70fd8ac2ff0e6efdc7e18310e6faa3dc85edfd5bc5d392fb23e6e344" Oct 30 05:00:03.042175 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.042148 4793 generic.go:356] "Generic (PLEG): container finished" podID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" containerID="3efdd84fe44aa60f00059ffb6e48036315ebaf050bcd34aca080d6f578a752bf" exitCode=60 Oct 30 05:00:03.042294 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.042226 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" event={"ID":"f0b116c3-eb39-4ffb-b765-666e2a9a9966","Type":"ContainerDied","Data":"3efdd84fe44aa60f00059ffb6e48036315ebaf050bcd34aca080d6f578a752bf"} Oct 30 05:00:03.042494 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.042478 4793 scope.go:122] "RemoveContainer" containerID="3efdd84fe44aa60f00059ffb6e48036315ebaf050bcd34aca080d6f578a752bf" Oct 30 05:00:03.044432 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.044395 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" event={"ID":"3090898b-f127-44a1-85c3-ca256c7cb642","Type":"ContainerStarted","Data":"521104cf4239048a7001e854129337d4312841171b9b3c6bde538168be347889"} Oct 30 05:00:03.046372 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.046247 4793 generic.go:356] "Generic (PLEG): container finished" podID="43b309b4-549a-48c2-9260-fc1ae51dd264" containerID="30ef2873478e6fbb687cbd32684d476d0defc96bdedaf87cae889fa7eba6a9ed" exitCode=60 Oct 30 05:00:03.046372 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.046326 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" event={"ID":"43b309b4-549a-48c2-9260-fc1ae51dd264","Type":"ContainerDied","Data":"30ef2873478e6fbb687cbd32684d476d0defc96bdedaf87cae889fa7eba6a9ed"} Oct 30 05:00:03.046612 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.046512 4793 scope.go:122] "RemoveContainer" containerID="30ef2873478e6fbb687cbd32684d476d0defc96bdedaf87cae889fa7eba6a9ed" Oct 30 05:00:03.048001 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.047980 4793 generic.go:356] "Generic (PLEG): container finished" podID="1a17c173-623d-4564-9f4c-d590922b6520" containerID="aa136c158357a505b0e3847ea09cdec9c5e4606ba4810848ba4b0cb934c47fa6" exitCode=60 Oct 30 05:00:03.048082 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.048060 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" event={"ID":"1a17c173-623d-4564-9f4c-d590922b6520","Type":"ContainerDied","Data":"aa136c158357a505b0e3847ea09cdec9c5e4606ba4810848ba4b0cb934c47fa6"} Oct 30 05:00:03.048248 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.048234 4793 scope.go:122] "RemoveContainer" containerID="aa136c158357a505b0e3847ea09cdec9c5e4606ba4810848ba4b0cb934c47fa6" Oct 30 05:00:03.049765 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.049746 4793 generic.go:356] "Generic (PLEG): container finished" podID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" containerID="5f1e9091fdb40b04d224b8338d1ab19630f570b34360a2c2d5a4298824872ce0" exitCode=60 Oct 30 05:00:03.049850 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.049824 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" event={"ID":"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8","Type":"ContainerDied","Data":"5f1e9091fdb40b04d224b8338d1ab19630f570b34360a2c2d5a4298824872ce0"} Oct 30 05:00:03.050019 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.050001 4793 scope.go:122] "RemoveContainer" containerID="5f1e9091fdb40b04d224b8338d1ab19630f570b34360a2c2d5a4298824872ce0" Oct 30 05:00:03.051557 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.051533 4793 generic.go:356] "Generic (PLEG): container finished" podID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" containerID="c709864a995754af1885ad10ef3bf71681bdc9d16c2568b9578310a126347498" exitCode=60 Oct 30 05:00:03.051667 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.051639 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" event={"ID":"d0225097-2d12-43e4-89b8-ea6b75c9ea54","Type":"ContainerDied","Data":"c709864a995754af1885ad10ef3bf71681bdc9d16c2568b9578310a126347498"} Oct 30 05:00:03.051852 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.051828 4793 scope.go:122] "RemoveContainer" containerID="c709864a995754af1885ad10ef3bf71681bdc9d16c2568b9578310a126347498" Oct 30 05:00:03.054072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.053395 4793 generic.go:356] "Generic (PLEG): container finished" podID="1f66a424-269e-475e-bf86-3e40027c9934" containerID="1a5494cfa829041cb3dbdb768b7c2b4623a62ca6c2c89a01ccf5bf442f54260a" exitCode=60 Oct 30 05:00:03.054072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.053464 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" event={"ID":"1f66a424-269e-475e-bf86-3e40027c9934","Type":"ContainerDied","Data":"1a5494cfa829041cb3dbdb768b7c2b4623a62ca6c2c89a01ccf5bf442f54260a"} Oct 30 05:00:03.054072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.053728 4793 scope.go:122] "RemoveContainer" containerID="1a5494cfa829041cb3dbdb768b7c2b4623a62ca6c2c89a01ccf5bf442f54260a" Oct 30 05:00:03.057379 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.057358 4793 generic.go:356] "Generic (PLEG): container finished" podID="291b86ef-89c7-4b35-b649-e90aa681670e" containerID="c577511a39ae0ea2a444699719735f2db853c98ad09c3e335f2db570eaa2d20b" exitCode=60 Oct 30 05:00:03.057479 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.057457 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" event={"ID":"291b86ef-89c7-4b35-b649-e90aa681670e","Type":"ContainerDied","Data":"c577511a39ae0ea2a444699719735f2db853c98ad09c3e335f2db570eaa2d20b"} Oct 30 05:00:03.057800 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.057778 4793 scope.go:122] "RemoveContainer" containerID="c577511a39ae0ea2a444699719735f2db853c98ad09c3e335f2db570eaa2d20b" Oct 30 05:00:03.059181 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.059159 4793 generic.go:356] "Generic (PLEG): container finished" podID="44d00032-6fea-4d6d-93d8-e481e477cc96" containerID="f2fa0e6fcb1c7a814bfa6856dd7e41ede9341a1da9f3cd8b03e515cf55a825c9" exitCode=60 Oct 30 05:00:03.059269 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.059200 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" event={"ID":"44d00032-6fea-4d6d-93d8-e481e477cc96","Type":"ContainerDied","Data":"f2fa0e6fcb1c7a814bfa6856dd7e41ede9341a1da9f3cd8b03e515cf55a825c9"} Oct 30 05:00:03.059475 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.059460 4793 scope.go:122] "RemoveContainer" containerID="f2fa0e6fcb1c7a814bfa6856dd7e41ede9341a1da9f3cd8b03e515cf55a825c9" Oct 30 05:00:03.293198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.293143 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" podStartSLOduration=3.293129179 podStartE2EDuration="3.293129179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:03.290798958 +0000 UTC m=+3726.576673729" watchObservedRunningTime="2026-10-30 05:00:03.293129179 +0000 UTC m=+3726.579003951" Oct 30 05:00:03.595052 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.595020 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc"] Oct 30 05:00:03.599921 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.599897 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:03.617645 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.617622 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-26b72f-serving-cert\"" Oct 30 05:00:03.619537 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.619512 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc"] Oct 30 05:00:03.654856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.654543 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2996a1cc-21e1-4025-bb05-a52f82f82958-openshift-service-ca\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:03.654856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.654643 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rb4z\" (UniqueName: \"kubernetes.io/projected/2996a1cc-21e1-4025-bb05-a52f82f82958-kube-api-access-4rb4z\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:03.654856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.654716 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-metrics-certs\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:03.655134 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.654946 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-maas-api-tls\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:03.758866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.757168 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-maas-api-tls\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:03.758866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.757302 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2996a1cc-21e1-4025-bb05-a52f82f82958-openshift-service-ca\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:03.758866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.757376 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rb4z\" (UniqueName: \"kubernetes.io/projected/2996a1cc-21e1-4025-bb05-a52f82f82958-kube-api-access-4rb4z\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:03.758866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.757462 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-metrics-certs\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:03.758866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.758833 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2996a1cc-21e1-4025-bb05-a52f82f82958-openshift-service-ca\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:03.759244 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:03.758933 4793 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-serving-cert: secret "maas-api-e2e-inf-b-26b72f-serving-cert" not found Oct 30 05:00:03.759244 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:03.758984 4793 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-maas-api-tls podName:2996a1cc-21e1-4025-bb05-a52f82f82958 nodeName:}" failed. No retries permitted until 2026-10-30 05:00:04.258967052 +0000 UTC m=+3727.544841813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-maas-api-tls") pod "maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" (UID: "2996a1cc-21e1-4025-bb05-a52f82f82958") : secret "maas-api-e2e-inf-b-26b72f-serving-cert" not found Oct 30 05:00:03.763054 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.762894 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-metrics-certs\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:03.810946 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.810050 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:03.842151 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:03.842113 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rb4z\" (UniqueName: \"kubernetes.io/projected/2996a1cc-21e1-4025-bb05-a52f82f82958-kube-api-access-4rb4z\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:04.072246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.072214 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" event={"ID":"54c38420-6bf8-4305-bbbf-6656194e1003","Type":"ContainerStarted","Data":"7da8ed6e82692186f1b1087e096ef856a39ee42c6d795f51e18670c648c21d94"} Oct 30 05:00:04.074232 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.074206 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" event={"ID":"f0b116c3-eb39-4ffb-b765-666e2a9a9966","Type":"ContainerStarted","Data":"ff8c0fd29188ee4ad2807446fbd5ee1163659869cb177a96aa55f4abea0e67cd"} Oct 30 05:00:04.076108 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.076080 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:04.076438 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.076402 4793 generic.go:356] "Generic (PLEG): container finished" podID="3090898b-f127-44a1-85c3-ca256c7cb642" containerID="521104cf4239048a7001e854129337d4312841171b9b3c6bde538168be347889" exitCode=60 Oct 30 05:00:04.076546 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.076468 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" event={"ID":"3090898b-f127-44a1-85c3-ca256c7cb642","Type":"ContainerDied","Data":"521104cf4239048a7001e854129337d4312841171b9b3c6bde538168be347889"} Oct 30 05:00:04.076759 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.076738 4793 scope.go:122] "RemoveContainer" containerID="521104cf4239048a7001e854129337d4312841171b9b3c6bde538168be347889" Oct 30 05:00:04.079131 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.079110 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" event={"ID":"43b309b4-549a-48c2-9260-fc1ae51dd264","Type":"ContainerStarted","Data":"c75377becbbc3d5f794326488bd3e89c1da12b1ef4597dd17f91328f0c976ec5"} Oct 30 05:00:04.084329 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.081716 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" event={"ID":"1a17c173-623d-4564-9f4c-d590922b6520","Type":"ContainerStarted","Data":"2e3e349bfdad4f80c1fb2e5e3128d1bf3de86b0203e00efe1cff07d0199fef6f"} Oct 30 05:00:04.086245 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.084675 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" event={"ID":"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8","Type":"ContainerStarted","Data":"bcda78fe17714d56b6cdb47fcfa0bb50949dd397cbc5edddc2d2bc54c7cff660"} Oct 30 05:00:04.088911 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.088886 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" event={"ID":"d0225097-2d12-43e4-89b8-ea6b75c9ea54","Type":"ContainerStarted","Data":"23d65343004b58969a93998420a7c50fe2aa257da25f805e85c8be8174c50e0a"} Oct 30 05:00:04.093323 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.093290 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" event={"ID":"1f66a424-269e-475e-bf86-3e40027c9934","Type":"ContainerStarted","Data":"0e8023599aec00770b6669abddb1f7ab51362fddf0e258347880ca99aadfe2e4"} Oct 30 05:00:04.095783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.095761 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" event={"ID":"291b86ef-89c7-4b35-b649-e90aa681670e","Type":"ContainerStarted","Data":"db01b1b6ef131dbebfe404a81bb4ff7d9e63af8828f8c83a6c27b8be00ec1e53"} Oct 30 05:00:04.098382 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.098277 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" event={"ID":"44d00032-6fea-4d6d-93d8-e481e477cc96","Type":"ContainerStarted","Data":"e14c5e341881543ec2c65e1546fa9850c9818e5eccbbad173428bc53bb386950"} Oct 30 05:00:04.130533 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.130491 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" podStartSLOduration=4.130479805 podStartE2EDuration="4.130479805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:04.126732746 +0000 UTC m=+3727.412607520" watchObservedRunningTime="2026-10-30 05:00:04.130479805 +0000 UTC m=+3727.416354580" Oct 30 05:00:04.173466 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.169164 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" podStartSLOduration=4.169149435 podStartE2EDuration="4.169149435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:04.16660134 +0000 UTC m=+3727.452476117" watchObservedRunningTime="2026-10-30 05:00:04.169149435 +0000 UTC m=+3727.455024210" Oct 30 05:00:04.211807 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.211762 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" podStartSLOduration=4.211748764 podStartE2EDuration="4.211748764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:04.206784267 +0000 UTC m=+3727.492659042" watchObservedRunningTime="2026-10-30 05:00:04.211748764 +0000 UTC m=+3727.497623538" Oct 30 05:00:04.249311 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.249270 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" podStartSLOduration=4.249258325 podStartE2EDuration="4.249258325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:04.248491217 +0000 UTC m=+3727.534365992" watchObservedRunningTime="2026-10-30 05:00:04.249258325 +0000 UTC m=+3727.535133099" Oct 30 05:00:04.273534 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.273502 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-maas-api-tls\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:04.276094 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.276069 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-maas-api-tls\") pod \"maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:04.320935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.320901 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:04.383130 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.383078 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" podStartSLOduration=4.38306346 podStartE2EDuration="4.38306346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:04.381183799 +0000 UTC m=+3727.667058595" watchObservedRunningTime="2026-10-30 05:00:04.38306346 +0000 UTC m=+3727.668938258" Oct 30 05:00:04.470630 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.470583 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" podStartSLOduration=4.470569061 podStartE2EDuration="4.470569061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:04.422140153 +0000 UTC m=+3727.708014928" watchObservedRunningTime="2026-10-30 05:00:04.470569061 +0000 UTC m=+3727.756444252" Oct 30 05:00:04.473599 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.473558 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" podStartSLOduration=4.473546446 podStartE2EDuration="4.473546446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:04.468279866 +0000 UTC m=+3727.754154631" watchObservedRunningTime="2026-10-30 05:00:04.473546446 +0000 UTC m=+3727.759421220" Oct 30 05:00:04.492671 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.492647 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:04.824836 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.824804 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc"] Oct 30 05:00:04.844060 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:00:04.844032 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2996a1cc_21e1_4025_bb05_a52f82f82958.slice/crio-070bb629916ae23f5f58f0cd3b392f005c66b080fe48c576eb3c20669d2e472a WatchSource:0}: Error finding container 070bb629916ae23f5f58f0cd3b392f005c66b080fe48c576eb3c20669d2e472a: Status 404 returned error can't find the container with id 070bb629916ae23f5f58f0cd3b392f005c66b080fe48c576eb3c20669d2e472a Oct 30 05:00:04.845587 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:04.845566 4793 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:00:05.110384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.110354 4793 generic.go:356] "Generic (PLEG): container finished" podID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" containerID="23d65343004b58969a93998420a7c50fe2aa257da25f805e85c8be8174c50e0a" exitCode=60 Oct 30 05:00:05.110845 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.110399 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" event={"ID":"d0225097-2d12-43e4-89b8-ea6b75c9ea54","Type":"ContainerDied","Data":"23d65343004b58969a93998420a7c50fe2aa257da25f805e85c8be8174c50e0a"} Oct 30 05:00:05.110845 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.110505 4793 scope.go:122] "RemoveContainer" containerID="c709864a995754af1885ad10ef3bf71681bdc9d16c2568b9578310a126347498" Oct 30 05:00:05.110845 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.110813 4793 scope.go:122] "RemoveContainer" containerID="23d65343004b58969a93998420a7c50fe2aa257da25f805e85c8be8174c50e0a" Oct 30 05:00:05.111222 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:05.111205 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf_odh-ai-gateway-infra(d0225097-2d12-43e4-89b8-ea6b75c9ea54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" Oct 30 05:00:05.112673 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.112645 4793 generic.go:356] "Generic (PLEG): container finished" podID="1f66a424-269e-475e-bf86-3e40027c9934" containerID="0e8023599aec00770b6669abddb1f7ab51362fddf0e258347880ca99aadfe2e4" exitCode=60 Oct 30 05:00:05.112779 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.112761 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" event={"ID":"1f66a424-269e-475e-bf86-3e40027c9934","Type":"ContainerDied","Data":"0e8023599aec00770b6669abddb1f7ab51362fddf0e258347880ca99aadfe2e4"} Oct 30 05:00:05.113042 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.113019 4793 scope.go:122] "RemoveContainer" containerID="0e8023599aec00770b6669abddb1f7ab51362fddf0e258347880ca99aadfe2e4" Oct 30 05:00:05.113413 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:05.113390 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57_odh-ai-gateway-infra(1f66a424-269e-475e-bf86-3e40027c9934)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" podUID="1f66a424-269e-475e-bf86-3e40027c9934" Oct 30 05:00:05.114663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.114644 4793 generic.go:356] "Generic (PLEG): container finished" podID="291b86ef-89c7-4b35-b649-e90aa681670e" containerID="db01b1b6ef131dbebfe404a81bb4ff7d9e63af8828f8c83a6c27b8be00ec1e53" exitCode=60 Oct 30 05:00:05.114753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.114731 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" event={"ID":"291b86ef-89c7-4b35-b649-e90aa681670e","Type":"ContainerDied","Data":"db01b1b6ef131dbebfe404a81bb4ff7d9e63af8828f8c83a6c27b8be00ec1e53"} Oct 30 05:00:05.114978 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.114957 4793 scope.go:122] "RemoveContainer" containerID="db01b1b6ef131dbebfe404a81bb4ff7d9e63af8828f8c83a6c27b8be00ec1e53" Oct 30 05:00:05.115310 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:05.115288 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz_odh-ai-gateway-infra(291b86ef-89c7-4b35-b649-e90aa681670e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" Oct 30 05:00:05.116746 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.116725 4793 generic.go:356] "Generic (PLEG): container finished" podID="44d00032-6fea-4d6d-93d8-e481e477cc96" containerID="e14c5e341881543ec2c65e1546fa9850c9818e5eccbbad173428bc53bb386950" exitCode=60 Oct 30 05:00:05.116839 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.116789 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" event={"ID":"44d00032-6fea-4d6d-93d8-e481e477cc96","Type":"ContainerDied","Data":"e14c5e341881543ec2c65e1546fa9850c9818e5eccbbad173428bc53bb386950"} Oct 30 05:00:05.116992 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.116974 4793 scope.go:122] "RemoveContainer" containerID="e14c5e341881543ec2c65e1546fa9850c9818e5eccbbad173428bc53bb386950" Oct 30 05:00:05.117290 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:05.117270 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42_odh-ai-gateway-infra(44d00032-6fea-4d6d-93d8-e481e477cc96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" Oct 30 05:00:05.118884 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.118862 4793 generic.go:356] "Generic (PLEG): container finished" podID="54c38420-6bf8-4305-bbbf-6656194e1003" containerID="7da8ed6e82692186f1b1087e096ef856a39ee42c6d795f51e18670c648c21d94" exitCode=60 Oct 30 05:00:05.118985 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.118895 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" event={"ID":"54c38420-6bf8-4305-bbbf-6656194e1003","Type":"ContainerDied","Data":"7da8ed6e82692186f1b1087e096ef856a39ee42c6d795f51e18670c648c21d94"} Oct 30 05:00:05.119141 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.119123 4793 scope.go:122] "RemoveContainer" containerID="7da8ed6e82692186f1b1087e096ef856a39ee42c6d795f51e18670c648c21d94" Oct 30 05:00:05.119428 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:05.119392 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn_odh-ai-gateway-infra(54c38420-6bf8-4305-bbbf-6656194e1003)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" Oct 30 05:00:05.120887 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.120865 4793 generic.go:356] "Generic (PLEG): container finished" podID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" containerID="ff8c0fd29188ee4ad2807446fbd5ee1163659869cb177a96aa55f4abea0e67cd" exitCode=60 Oct 30 05:00:05.120992 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.120942 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" event={"ID":"f0b116c3-eb39-4ffb-b765-666e2a9a9966","Type":"ContainerDied","Data":"ff8c0fd29188ee4ad2807446fbd5ee1163659869cb177a96aa55f4abea0e67cd"} Oct 30 05:00:05.121156 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.121137 4793 scope.go:122] "RemoveContainer" containerID="ff8c0fd29188ee4ad2807446fbd5ee1163659869cb177a96aa55f4abea0e67cd" Oct 30 05:00:05.121432 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:05.121395 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5_odh-ai-gateway-infra(f0b116c3-eb39-4ffb-b765-666e2a9a9966)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" Oct 30 05:00:05.123017 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.122994 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" event={"ID":"3090898b-f127-44a1-85c3-ca256c7cb642","Type":"ContainerStarted","Data":"647ddd88b078ecdaa491c37326cb9eef638837f9a1411f932b5f689f6f38a616"} Oct 30 05:00:05.124821 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.124802 4793 generic.go:356] "Generic (PLEG): container finished" podID="43b309b4-549a-48c2-9260-fc1ae51dd264" containerID="c75377becbbc3d5f794326488bd3e89c1da12b1ef4597dd17f91328f0c976ec5" exitCode=60 Oct 30 05:00:05.124920 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.124823 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" event={"ID":"43b309b4-549a-48c2-9260-fc1ae51dd264","Type":"ContainerDied","Data":"c75377becbbc3d5f794326488bd3e89c1da12b1ef4597dd17f91328f0c976ec5"} Oct 30 05:00:05.125157 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.125135 4793 scope.go:122] "RemoveContainer" containerID="c75377becbbc3d5f794326488bd3e89c1da12b1ef4597dd17f91328f0c976ec5" Oct 30 05:00:05.125522 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:05.125495 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4_odh-ai-gateway-infra(43b309b4-549a-48c2-9260-fc1ae51dd264)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" Oct 30 05:00:05.126027 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.126009 4793 scope.go:122] "RemoveContainer" containerID="1a5494cfa829041cb3dbdb768b7c2b4623a62ca6c2c89a01ccf5bf442f54260a" Oct 30 05:00:05.126889 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.126871 4793 generic.go:356] "Generic (PLEG): container finished" podID="1a17c173-623d-4564-9f4c-d590922b6520" containerID="2e3e349bfdad4f80c1fb2e5e3128d1bf3de86b0203e00efe1cff07d0199fef6f" exitCode=60 Oct 30 05:00:05.126982 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.126952 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" event={"ID":"1a17c173-623d-4564-9f4c-d590922b6520","Type":"ContainerDied","Data":"2e3e349bfdad4f80c1fb2e5e3128d1bf3de86b0203e00efe1cff07d0199fef6f"} Oct 30 05:00:05.127154 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.127137 4793 scope.go:122] "RemoveContainer" containerID="2e3e349bfdad4f80c1fb2e5e3128d1bf3de86b0203e00efe1cff07d0199fef6f" Oct 30 05:00:05.127394 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:05.127370 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn_odh-ai-gateway-infra(1a17c173-623d-4564-9f4c-d590922b6520)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" podUID="1a17c173-623d-4564-9f4c-d590922b6520" Oct 30 05:00:05.128981 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.128964 4793 generic.go:356] "Generic (PLEG): container finished" podID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" containerID="bcda78fe17714d56b6cdb47fcfa0bb50949dd397cbc5edddc2d2bc54c7cff660" exitCode=60 Oct 30 05:00:05.129067 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.129039 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" event={"ID":"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8","Type":"ContainerDied","Data":"bcda78fe17714d56b6cdb47fcfa0bb50949dd397cbc5edddc2d2bc54c7cff660"} Oct 30 05:00:05.129246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.129228 4793 scope.go:122] "RemoveContainer" containerID="bcda78fe17714d56b6cdb47fcfa0bb50949dd397cbc5edddc2d2bc54c7cff660" Oct 30 05:00:05.129534 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:05.129510 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb_odh-ai-gateway-infra(cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" Oct 30 05:00:05.130326 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.130310 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" event={"ID":"2996a1cc-21e1-4025-bb05-a52f82f82958","Type":"ContainerStarted","Data":"070bb629916ae23f5f58f0cd3b392f005c66b080fe48c576eb3c20669d2e472a"} Oct 30 05:00:05.141169 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.141151 4793 scope.go:122] "RemoveContainer" containerID="c577511a39ae0ea2a444699719735f2db853c98ad09c3e335f2db570eaa2d20b" Oct 30 05:00:05.157553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.157490 4793 scope.go:122] "RemoveContainer" containerID="f2fa0e6fcb1c7a814bfa6856dd7e41ede9341a1da9f3cd8b03e515cf55a825c9" Oct 30 05:00:05.175844 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.175815 4793 scope.go:122] "RemoveContainer" containerID="fb9e815c70fd8ac2ff0e6efdc7e18310e6faa3dc85edfd5bc5d392fb23e6e344" Oct 30 05:00:05.191631 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.191610 4793 scope.go:122] "RemoveContainer" containerID="3efdd84fe44aa60f00059ffb6e48036315ebaf050bcd34aca080d6f578a752bf" Oct 30 05:00:05.206410 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.206391 4793 scope.go:122] "RemoveContainer" containerID="30ef2873478e6fbb687cbd32684d476d0defc96bdedaf87cae889fa7eba6a9ed" Oct 30 05:00:05.228175 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.228159 4793 scope.go:122] "RemoveContainer" containerID="aa136c158357a505b0e3847ea09cdec9c5e4606ba4810848ba4b0cb934c47fa6" Oct 30 05:00:05.249564 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.249543 4793 scope.go:122] "RemoveContainer" containerID="5f1e9091fdb40b04d224b8338d1ab19630f570b34360a2c2d5a4298824872ce0" Oct 30 05:00:05.907012 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:05.899721 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:06.142050 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.141965 4793 scope.go:122] "RemoveContainer" containerID="0e8023599aec00770b6669abddb1f7ab51362fddf0e258347880ca99aadfe2e4" Oct 30 05:00:06.142483 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:06.142346 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57_odh-ai-gateway-infra(1f66a424-269e-475e-bf86-3e40027c9934)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" podUID="1f66a424-269e-475e-bf86-3e40027c9934" Oct 30 05:00:06.143754 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.143729 4793 scope.go:122] "RemoveContainer" containerID="db01b1b6ef131dbebfe404a81bb4ff7d9e63af8828f8c83a6c27b8be00ec1e53" Oct 30 05:00:06.144100 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:06.144071 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz_odh-ai-gateway-infra(291b86ef-89c7-4b35-b649-e90aa681670e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" Oct 30 05:00:06.145477 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.145455 4793 scope.go:122] "RemoveContainer" containerID="e14c5e341881543ec2c65e1546fa9850c9818e5eccbbad173428bc53bb386950" Oct 30 05:00:06.145786 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:06.145762 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42_odh-ai-gateway-infra(44d00032-6fea-4d6d-93d8-e481e477cc96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" Oct 30 05:00:06.147223 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.147206 4793 scope.go:122] "RemoveContainer" containerID="7da8ed6e82692186f1b1087e096ef856a39ee42c6d795f51e18670c648c21d94" Oct 30 05:00:06.147608 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:06.147583 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn_odh-ai-gateway-infra(54c38420-6bf8-4305-bbbf-6656194e1003)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" Oct 30 05:00:06.148919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.148900 4793 scope.go:122] "RemoveContainer" containerID="ff8c0fd29188ee4ad2807446fbd5ee1163659869cb177a96aa55f4abea0e67cd" Oct 30 05:00:06.149167 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:06.149143 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5_odh-ai-gateway-infra(f0b116c3-eb39-4ffb-b765-666e2a9a9966)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" Oct 30 05:00:06.150627 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.150601 4793 generic.go:356] "Generic (PLEG): container finished" podID="3090898b-f127-44a1-85c3-ca256c7cb642" containerID="647ddd88b078ecdaa491c37326cb9eef638837f9a1411f932b5f689f6f38a616" exitCode=60 Oct 30 05:00:06.150735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.150682 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" event={"ID":"3090898b-f127-44a1-85c3-ca256c7cb642","Type":"ContainerDied","Data":"647ddd88b078ecdaa491c37326cb9eef638837f9a1411f932b5f689f6f38a616"} Oct 30 05:00:06.150735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.150726 4793 scope.go:122] "RemoveContainer" containerID="521104cf4239048a7001e854129337d4312841171b9b3c6bde538168be347889" Oct 30 05:00:06.150894 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.150875 4793 scope.go:122] "RemoveContainer" containerID="647ddd88b078ecdaa491c37326cb9eef638837f9a1411f932b5f689f6f38a616" Oct 30 05:00:06.151153 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:06.151130 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b_odh-ai-gateway-infra(3090898b-f127-44a1-85c3-ca256c7cb642)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" Oct 30 05:00:06.152626 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.152609 4793 scope.go:122] "RemoveContainer" containerID="c75377becbbc3d5f794326488bd3e89c1da12b1ef4597dd17f91328f0c976ec5" Oct 30 05:00:06.152862 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:06.152843 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4_odh-ai-gateway-infra(43b309b4-549a-48c2-9260-fc1ae51dd264)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" Oct 30 05:00:06.154345 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.154328 4793 scope.go:122] "RemoveContainer" containerID="2e3e349bfdad4f80c1fb2e5e3128d1bf3de86b0203e00efe1cff07d0199fef6f" Oct 30 05:00:06.154772 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:06.154729 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn_odh-ai-gateway-infra(1a17c173-623d-4564-9f4c-d590922b6520)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" podUID="1a17c173-623d-4564-9f4c-d590922b6520" Oct 30 05:00:06.156580 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.156552 4793 scope.go:122] "RemoveContainer" containerID="bcda78fe17714d56b6cdb47fcfa0bb50949dd397cbc5edddc2d2bc54c7cff660" Oct 30 05:00:06.156864 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:06.156829 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb_odh-ai-gateway-infra(cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" Oct 30 05:00:06.158064 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.158043 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" event={"ID":"2996a1cc-21e1-4025-bb05-a52f82f82958","Type":"ContainerStarted","Data":"1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430"} Oct 30 05:00:06.158194 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.158176 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:06.160359 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.160342 4793 scope.go:122] "RemoveContainer" containerID="23d65343004b58969a93998420a7c50fe2aa257da25f805e85c8be8174c50e0a" Oct 30 05:00:06.160634 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:06.160614 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf_odh-ai-gateway-infra(d0225097-2d12-43e4-89b8-ea6b75c9ea54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" Oct 30 05:00:06.597602 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:06.597538 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" podStartSLOduration=3.1851407 podStartE2EDuration="3.597519605s" totalImagesPullingTime="412.378905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:06.58944334 +0000 UTC m=+3729.875318144" watchObservedRunningTime="2026-10-30 05:00:06.597519605 +0000 UTC m=+3729.883394382" Oct 30 05:00:07.174813 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:07.174782 4793 scope.go:122] "RemoveContainer" containerID="647ddd88b078ecdaa491c37326cb9eef638837f9a1411f932b5f689f6f38a616" Oct 30 05:00:07.175260 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:07.175234 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b_odh-ai-gateway-infra(3090898b-f127-44a1-85c3-ca256c7cb642)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" Oct 30 05:00:07.773032 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:07.772994 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:09.536493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:09.536030 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:11.222095 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:11.222053 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:11.236232 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:11.236187 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:00:12.180964 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:12.180932 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:00:12.891925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:12.891892 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:14.539235 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:14.539187 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:15.838299 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:15.838270 4793 scope.go:122] "RemoveContainer" containerID="db01b1b6ef131dbebfe404a81bb4ff7d9e63af8828f8c83a6c27b8be00ec1e53" Oct 30 05:00:16.256174 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:16.256127 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:16.845497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:16.844894 4793 scope.go:122] "RemoveContainer" containerID="7da8ed6e82692186f1b1087e096ef856a39ee42c6d795f51e18670c648c21d94" Oct 30 05:00:16.845497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:16.845300 4793 scope.go:122] "RemoveContainer" containerID="ff8c0fd29188ee4ad2807446fbd5ee1163659869cb177a96aa55f4abea0e67cd" Oct 30 05:00:16.846466 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:16.846119 4793 scope.go:122] "RemoveContainer" containerID="23d65343004b58969a93998420a7c50fe2aa257da25f805e85c8be8174c50e0a" Oct 30 05:00:16.846547 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:16.846470 4793 scope.go:122] "RemoveContainer" containerID="647ddd88b078ecdaa491c37326cb9eef638837f9a1411f932b5f689f6f38a616" Oct 30 05:00:16.846547 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:16.846519 4793 scope.go:122] "RemoveContainer" containerID="0e8023599aec00770b6669abddb1f7ab51362fddf0e258347880ca99aadfe2e4" Oct 30 05:00:16.847367 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:16.847288 4793 scope.go:122] "RemoveContainer" containerID="2e3e349bfdad4f80c1fb2e5e3128d1bf3de86b0203e00efe1cff07d0199fef6f" Oct 30 05:00:17.272960 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:17.272927 4793 generic.go:356] "Generic (PLEG): container finished" podID="291b86ef-89c7-4b35-b649-e90aa681670e" containerID="8a0d453e981e5827eb5973d191448e437958d43cec9077dd23df92d5d225a702" exitCode=60 Oct 30 05:00:17.273154 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:17.273000 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" event={"ID":"291b86ef-89c7-4b35-b649-e90aa681670e","Type":"ContainerDied","Data":"8a0d453e981e5827eb5973d191448e437958d43cec9077dd23df92d5d225a702"} Oct 30 05:00:17.273154 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:17.273074 4793 scope.go:122] "RemoveContainer" containerID="db01b1b6ef131dbebfe404a81bb4ff7d9e63af8828f8c83a6c27b8be00ec1e53" Oct 30 05:00:17.273696 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:17.273636 4793 scope.go:122] "RemoveContainer" containerID="8a0d453e981e5827eb5973d191448e437958d43cec9077dd23df92d5d225a702" Oct 30 05:00:17.274143 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:17.274064 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz_odh-ai-gateway-infra(291b86ef-89c7-4b35-b649-e90aa681670e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" Oct 30 05:00:17.838277 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:17.838213 4793 scope.go:122] "RemoveContainer" containerID="c75377becbbc3d5f794326488bd3e89c1da12b1ef4597dd17f91328f0c976ec5" Oct 30 05:00:17.838500 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:17.838480 4793 scope.go:122] "RemoveContainer" containerID="e14c5e341881543ec2c65e1546fa9850c9818e5eccbbad173428bc53bb386950" Oct 30 05:00:18.004014 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.003957 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:18.285860 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.285821 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" event={"ID":"54c38420-6bf8-4305-bbbf-6656194e1003","Type":"ContainerStarted","Data":"f22de6650b57e86d58d7a88b73e7114b72c3f31acace7d075c98cf116b423efa"} Oct 30 05:00:18.288146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.288115 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" event={"ID":"f0b116c3-eb39-4ffb-b765-666e2a9a9966","Type":"ContainerStarted","Data":"c1d7afe51fc348f74c4a0b9a51095fbc9167118f43e9cc85d2a757470c30615f"} Oct 30 05:00:18.290324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.290297 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" event={"ID":"3090898b-f127-44a1-85c3-ca256c7cb642","Type":"ContainerStarted","Data":"653a306dee99b4c9d764752995fbc84c4eb58e9a0f4b9e9beed9d3166dff7de2"} Oct 30 05:00:18.292688 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.292658 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" event={"ID":"1a17c173-623d-4564-9f4c-d590922b6520","Type":"ContainerStarted","Data":"0a96021c8292b19fb81454307048fcd45316974c6d12a7b1b81af6ea2aabd732"} Oct 30 05:00:18.294745 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.294721 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" event={"ID":"d0225097-2d12-43e4-89b8-ea6b75c9ea54","Type":"ContainerStarted","Data":"44fbee2b30b8d60d50d9028b11da3c5c66dc41c0bed4d99985fddda4df99842c"} Oct 30 05:00:18.296857 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.296821 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" event={"ID":"1f66a424-269e-475e-bf86-3e40027c9934","Type":"ContainerStarted","Data":"a3ec6ca0db5275e9dff2a31f8e964c28839cfa89b24226a51eb494898e6889e0"} Oct 30 05:00:18.328822 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.328795 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz"] Oct 30 05:00:18.698444 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.698359 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" Oct 30 05:00:18.744331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.744293 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291b86ef-89c7-4b35-b649-e90aa681670e-kube-api-access-lxtj5\" (UniqueName: \"kubernetes.io/projected/291b86ef-89c7-4b35-b649-e90aa681670e-kube-api-access-lxtj5\") pod \"291b86ef-89c7-4b35-b649-e90aa681670e\" (UID: \"291b86ef-89c7-4b35-b649-e90aa681670e\") " Oct 30 05:00:18.744948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.744503 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/291b86ef-89c7-4b35-b649-e90aa681670e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291b86ef-89c7-4b35-b649-e90aa681670e-maas-api-service-ca\") pod \"291b86ef-89c7-4b35-b649-e90aa681670e\" (UID: \"291b86ef-89c7-4b35-b649-e90aa681670e\") " Oct 30 05:00:18.744948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.744908 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291b86ef-89c7-4b35-b649-e90aa681670e-maas-api-service-ca" pod "291b86ef-89c7-4b35-b649-e90aa681670e" (UID: "291b86ef-89c7-4b35-b649-e90aa681670e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:18.745320 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.745293 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/291b86ef-89c7-4b35-b649-e90aa681670e-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:18.747581 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.747553 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291b86ef-89c7-4b35-b649-e90aa681670e-kube-api-access-lxtj5" pod "291b86ef-89c7-4b35-b649-e90aa681670e" (UID: "291b86ef-89c7-4b35-b649-e90aa681670e"). InnerVolumeSpecName "kube-api-access-lxtj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:18.839174 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.839131 4793 scope.go:122] "RemoveContainer" containerID="bcda78fe17714d56b6cdb47fcfa0bb50949dd397cbc5edddc2d2bc54c7cff660" Oct 30 05:00:18.846560 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:18.846531 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxtj5\" (UniqueName: \"kubernetes.io/projected/291b86ef-89c7-4b35-b649-e90aa681670e-kube-api-access-lxtj5\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:19.311862 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.311795 4793 generic.go:356] "Generic (PLEG): container finished" podID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" containerID="44fbee2b30b8d60d50d9028b11da3c5c66dc41c0bed4d99985fddda4df99842c" exitCode=60 Oct 30 05:00:19.312276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.311847 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" event={"ID":"d0225097-2d12-43e4-89b8-ea6b75c9ea54","Type":"ContainerDied","Data":"44fbee2b30b8d60d50d9028b11da3c5c66dc41c0bed4d99985fddda4df99842c"} Oct 30 05:00:19.312276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.311915 4793 scope.go:122] "RemoveContainer" containerID="23d65343004b58969a93998420a7c50fe2aa257da25f805e85c8be8174c50e0a" Oct 30 05:00:19.312376 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.312334 4793 scope.go:122] "RemoveContainer" containerID="44fbee2b30b8d60d50d9028b11da3c5c66dc41c0bed4d99985fddda4df99842c" Oct 30 05:00:19.312815 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:19.312783 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf_odh-ai-gateway-infra(d0225097-2d12-43e4-89b8-ea6b75c9ea54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" Oct 30 05:00:19.314107 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.314085 4793 generic.go:356] "Generic (PLEG): container finished" podID="1f66a424-269e-475e-bf86-3e40027c9934" containerID="a3ec6ca0db5275e9dff2a31f8e964c28839cfa89b24226a51eb494898e6889e0" exitCode=60 Oct 30 05:00:19.314217 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.314158 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" event={"ID":"1f66a424-269e-475e-bf86-3e40027c9934","Type":"ContainerDied","Data":"a3ec6ca0db5275e9dff2a31f8e964c28839cfa89b24226a51eb494898e6889e0"} Oct 30 05:00:19.314611 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.314589 4793 scope.go:122] "RemoveContainer" containerID="a3ec6ca0db5275e9dff2a31f8e964c28839cfa89b24226a51eb494898e6889e0" Oct 30 05:00:19.314952 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:19.314929 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57_odh-ai-gateway-infra(1f66a424-269e-475e-bf86-3e40027c9934)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" podUID="1f66a424-269e-475e-bf86-3e40027c9934" Oct 30 05:00:19.315658 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.315644 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" Oct 30 05:00:19.315753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.315675 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz" event={"ID":"291b86ef-89c7-4b35-b649-e90aa681670e","Type":"ContainerDied","Data":"5f7b87fe5c032c0891fb3a1d65ef7b6f4c0b8b5c56c9998ec503395f58c51a3f"} Oct 30 05:00:19.317916 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.317891 4793 generic.go:356] "Generic (PLEG): container finished" podID="44d00032-6fea-4d6d-93d8-e481e477cc96" containerID="0a239380d37009ea41934116bc335866b60b29054912dec127acd050de665cd8" exitCode=60 Oct 30 05:00:19.318018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.317956 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" event={"ID":"44d00032-6fea-4d6d-93d8-e481e477cc96","Type":"ContainerDied","Data":"0a239380d37009ea41934116bc335866b60b29054912dec127acd050de665cd8"} Oct 30 05:00:19.318282 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.318263 4793 scope.go:122] "RemoveContainer" containerID="0a239380d37009ea41934116bc335866b60b29054912dec127acd050de665cd8" Oct 30 05:00:19.318601 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:19.318580 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42_odh-ai-gateway-infra(44d00032-6fea-4d6d-93d8-e481e477cc96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" Oct 30 05:00:19.320049 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.320029 4793 generic.go:356] "Generic (PLEG): container finished" podID="54c38420-6bf8-4305-bbbf-6656194e1003" containerID="f22de6650b57e86d58d7a88b73e7114b72c3f31acace7d075c98cf116b423efa" exitCode=60 Oct 30 05:00:19.320121 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.320100 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" event={"ID":"54c38420-6bf8-4305-bbbf-6656194e1003","Type":"ContainerDied","Data":"f22de6650b57e86d58d7a88b73e7114b72c3f31acace7d075c98cf116b423efa"} Oct 30 05:00:19.320381 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.320367 4793 scope.go:122] "RemoveContainer" containerID="f22de6650b57e86d58d7a88b73e7114b72c3f31acace7d075c98cf116b423efa" Oct 30 05:00:19.320650 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:19.320631 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn_odh-ai-gateway-infra(54c38420-6bf8-4305-bbbf-6656194e1003)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" Oct 30 05:00:19.321977 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.321956 4793 generic.go:356] "Generic (PLEG): container finished" podID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" containerID="c1d7afe51fc348f74c4a0b9a51095fbc9167118f43e9cc85d2a757470c30615f" exitCode=60 Oct 30 05:00:19.322074 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.322039 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" event={"ID":"f0b116c3-eb39-4ffb-b765-666e2a9a9966","Type":"ContainerDied","Data":"c1d7afe51fc348f74c4a0b9a51095fbc9167118f43e9cc85d2a757470c30615f"} Oct 30 05:00:19.322410 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.322384 4793 scope.go:122] "RemoveContainer" containerID="c1d7afe51fc348f74c4a0b9a51095fbc9167118f43e9cc85d2a757470c30615f" Oct 30 05:00:19.322674 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:19.322651 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5_odh-ai-gateway-infra(f0b116c3-eb39-4ffb-b765-666e2a9a9966)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" Oct 30 05:00:19.324117 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.324099 4793 generic.go:356] "Generic (PLEG): container finished" podID="3090898b-f127-44a1-85c3-ca256c7cb642" containerID="653a306dee99b4c9d764752995fbc84c4eb58e9a0f4b9e9beed9d3166dff7de2" exitCode=60 Oct 30 05:00:19.324209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.324159 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" event={"ID":"3090898b-f127-44a1-85c3-ca256c7cb642","Type":"ContainerDied","Data":"653a306dee99b4c9d764752995fbc84c4eb58e9a0f4b9e9beed9d3166dff7de2"} Oct 30 05:00:19.324513 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.324494 4793 scope.go:122] "RemoveContainer" containerID="653a306dee99b4c9d764752995fbc84c4eb58e9a0f4b9e9beed9d3166dff7de2" Oct 30 05:00:19.324763 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:19.324743 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b_odh-ai-gateway-infra(3090898b-f127-44a1-85c3-ca256c7cb642)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" Oct 30 05:00:19.326368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.326348 4793 generic.go:356] "Generic (PLEG): container finished" podID="43b309b4-549a-48c2-9260-fc1ae51dd264" containerID="88b59e505595f2bf1c3a29c025c05091021717b148404ceec5e2361d493b9896" exitCode=60 Oct 30 05:00:19.326537 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.326456 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" event={"ID":"43b309b4-549a-48c2-9260-fc1ae51dd264","Type":"ContainerDied","Data":"88b59e505595f2bf1c3a29c025c05091021717b148404ceec5e2361d493b9896"} Oct 30 05:00:19.326829 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.326804 4793 scope.go:122] "RemoveContainer" containerID="88b59e505595f2bf1c3a29c025c05091021717b148404ceec5e2361d493b9896" Oct 30 05:00:19.327124 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:19.327085 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4_odh-ai-gateway-infra(43b309b4-549a-48c2-9260-fc1ae51dd264)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" Oct 30 05:00:19.329355 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.329335 4793 generic.go:356] "Generic (PLEG): container finished" podID="1a17c173-623d-4564-9f4c-d590922b6520" containerID="0a96021c8292b19fb81454307048fcd45316974c6d12a7b1b81af6ea2aabd732" exitCode=60 Oct 30 05:00:19.329487 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.329395 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" event={"ID":"1a17c173-623d-4564-9f4c-d590922b6520","Type":"ContainerDied","Data":"0a96021c8292b19fb81454307048fcd45316974c6d12a7b1b81af6ea2aabd732"} Oct 30 05:00:19.329769 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.329740 4793 scope.go:122] "RemoveContainer" containerID="0a96021c8292b19fb81454307048fcd45316974c6d12a7b1b81af6ea2aabd732" Oct 30 05:00:19.330003 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:19.329982 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn_odh-ai-gateway-infra(1a17c173-623d-4564-9f4c-d590922b6520)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" podUID="1a17c173-623d-4564-9f4c-d590922b6520" Oct 30 05:00:19.339504 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.339486 4793 scope.go:122] "RemoveContainer" containerID="0e8023599aec00770b6669abddb1f7ab51362fddf0e258347880ca99aadfe2e4" Oct 30 05:00:19.344096 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.344074 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn"] Oct 30 05:00:19.354979 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.354956 4793 scope.go:122] "RemoveContainer" containerID="8a0d453e981e5827eb5973d191448e437958d43cec9077dd23df92d5d225a702" Oct 30 05:00:19.368751 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.368734 4793 scope.go:122] "RemoveContainer" containerID="e14c5e341881543ec2c65e1546fa9850c9818e5eccbbad173428bc53bb386950" Oct 30 05:00:19.385563 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.385536 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn"] Oct 30 05:00:19.386256 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.386236 4793 scope.go:122] "RemoveContainer" containerID="7da8ed6e82692186f1b1087e096ef856a39ee42c6d795f51e18670c648c21d94" Oct 30 05:00:19.403943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.403926 4793 scope.go:122] "RemoveContainer" containerID="ff8c0fd29188ee4ad2807446fbd5ee1163659869cb177a96aa55f4abea0e67cd" Oct 30 05:00:19.418324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.418308 4793 scope.go:122] "RemoveContainer" containerID="647ddd88b078ecdaa491c37326cb9eef638837f9a1411f932b5f689f6f38a616" Oct 30 05:00:19.429025 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.429000 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5"] Oct 30 05:00:19.431787 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.431770 4793 scope.go:122] "RemoveContainer" containerID="c75377becbbc3d5f794326488bd3e89c1da12b1ef4597dd17f91328f0c976ec5" Oct 30 05:00:19.446038 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.446022 4793 scope.go:122] "RemoveContainer" containerID="2e3e349bfdad4f80c1fb2e5e3128d1bf3de86b0203e00efe1cff07d0199fef6f" Oct 30 05:00:19.469231 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.469211 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57"] Oct 30 05:00:19.510351 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.510326 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b"] Oct 30 05:00:19.557181 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.557151 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf"] Oct 30 05:00:19.706439 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.703064 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz"] Oct 30 05:00:19.714858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.714824 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-247158-29888940-l4mxz"] Oct 30 05:00:19.773139 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:19.773103 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:20.358710 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.358608 4793 generic.go:356] "Generic (PLEG): container finished" podID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" containerID="cae9f6eccaae48bb5f1830a0324d57268782032a8452ad254de55dbf9aab5965" exitCode=60 Oct 30 05:00:20.358710 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.358680 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" event={"ID":"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8","Type":"ContainerDied","Data":"cae9f6eccaae48bb5f1830a0324d57268782032a8452ad254de55dbf9aab5965"} Oct 30 05:00:20.359249 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.358777 4793 scope.go:122] "RemoveContainer" containerID="bcda78fe17714d56b6cdb47fcfa0bb50949dd397cbc5edddc2d2bc54c7cff660" Oct 30 05:00:20.359328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.359281 4793 scope.go:122] "RemoveContainer" containerID="cae9f6eccaae48bb5f1830a0324d57268782032a8452ad254de55dbf9aab5965" Oct 30 05:00:20.359777 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:20.359737 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb_odh-ai-gateway-infra(cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" Oct 30 05:00:20.552099 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.549486 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4"] Oct 30 05:00:20.657508 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.647824 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42"] Oct 30 05:00:20.657508 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.653526 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" Oct 30 05:00:20.791361 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.790891 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f66a424-269e-475e-bf86-3e40027c9934-kube-api-access-9zz7c\" (UniqueName: \"kubernetes.io/projected/1f66a424-269e-475e-bf86-3e40027c9934-kube-api-access-9zz7c\") pod \"1f66a424-269e-475e-bf86-3e40027c9934\" (UID: \"1f66a424-269e-475e-bf86-3e40027c9934\") " Oct 30 05:00:20.791726 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.791686 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f66a424-269e-475e-bf86-3e40027c9934-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f66a424-269e-475e-bf86-3e40027c9934-maas-api-service-ca\") pod \"1f66a424-269e-475e-bf86-3e40027c9934\" (UID: \"1f66a424-269e-475e-bf86-3e40027c9934\") " Oct 30 05:00:20.792127 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.792101 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f66a424-269e-475e-bf86-3e40027c9934-maas-api-service-ca" pod "1f66a424-269e-475e-bf86-3e40027c9934" (UID: "1f66a424-269e-475e-bf86-3e40027c9934"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:20.793152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.793127 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f66a424-269e-475e-bf86-3e40027c9934-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:20.795940 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.795863 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" Oct 30 05:00:20.798663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.798639 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f66a424-269e-475e-bf86-3e40027c9934-kube-api-access-9zz7c" pod "1f66a424-269e-475e-bf86-3e40027c9934" (UID: "1f66a424-269e-475e-bf86-3e40027c9934"). InnerVolumeSpecName "kube-api-access-9zz7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:20.810973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.810943 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" Oct 30 05:00:20.881689 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.863162 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" path="/var/lib/kubelet/pods/291b86ef-89c7-4b35-b649-e90aa681670e/volumes" Oct 30 05:00:20.909919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.897112 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0225097-2d12-43e4-89b8-ea6b75c9ea54-kube-api-access-pjfls\" (UniqueName: \"kubernetes.io/projected/d0225097-2d12-43e4-89b8-ea6b75c9ea54-kube-api-access-pjfls\") pod \"d0225097-2d12-43e4-89b8-ea6b75c9ea54\" (UID: \"d0225097-2d12-43e4-89b8-ea6b75c9ea54\") " Oct 30 05:00:20.909919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.898920 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b116c3-eb39-4ffb-b765-666e2a9a9966-kube-api-access-zpkcs\" (UniqueName: \"kubernetes.io/projected/f0b116c3-eb39-4ffb-b765-666e2a9a9966-kube-api-access-zpkcs\") pod \"f0b116c3-eb39-4ffb-b765-666e2a9a9966\" (UID: \"f0b116c3-eb39-4ffb-b765-666e2a9a9966\") " Oct 30 05:00:20.909919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.898981 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0225097-2d12-43e4-89b8-ea6b75c9ea54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0225097-2d12-43e4-89b8-ea6b75c9ea54-maas-api-service-ca\") pod \"d0225097-2d12-43e4-89b8-ea6b75c9ea54\" (UID: \"d0225097-2d12-43e4-89b8-ea6b75c9ea54\") " Oct 30 05:00:20.909919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.899090 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b116c3-eb39-4ffb-b765-666e2a9a9966-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b116c3-eb39-4ffb-b765-666e2a9a9966-maas-api-service-ca\") pod \"f0b116c3-eb39-4ffb-b765-666e2a9a9966\" (UID: \"f0b116c3-eb39-4ffb-b765-666e2a9a9966\") " Oct 30 05:00:20.909919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.901092 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zz7c\" (UniqueName: \"kubernetes.io/projected/1f66a424-269e-475e-bf86-3e40027c9934-kube-api-access-9zz7c\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:20.909919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.902582 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b116c3-eb39-4ffb-b765-666e2a9a9966-maas-api-service-ca" pod "f0b116c3-eb39-4ffb-b765-666e2a9a9966" (UID: "f0b116c3-eb39-4ffb-b765-666e2a9a9966"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:20.912259 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.912218 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0225097-2d12-43e4-89b8-ea6b75c9ea54-maas-api-service-ca" pod "d0225097-2d12-43e4-89b8-ea6b75c9ea54" (UID: "d0225097-2d12-43e4-89b8-ea6b75c9ea54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:20.932231 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.930035 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" Oct 30 05:00:20.944833 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.944687 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b116c3-eb39-4ffb-b765-666e2a9a9966-kube-api-access-zpkcs" pod "f0b116c3-eb39-4ffb-b765-666e2a9a9966" (UID: "f0b116c3-eb39-4ffb-b765-666e2a9a9966"). InnerVolumeSpecName "kube-api-access-zpkcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:20.945952 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.945521 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0225097-2d12-43e4-89b8-ea6b75c9ea54-kube-api-access-pjfls" pod "d0225097-2d12-43e4-89b8-ea6b75c9ea54" (UID: "d0225097-2d12-43e4-89b8-ea6b75c9ea54"). InnerVolumeSpecName "kube-api-access-pjfls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:20.956488 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:20.956466 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" Oct 30 05:00:21.014273 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.013776 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" Oct 30 05:00:21.015146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.015113 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a17c173-623d-4564-9f4c-d590922b6520-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a17c173-623d-4564-9f4c-d590922b6520-maas-api-service-ca\") pod \"1a17c173-623d-4564-9f4c-d590922b6520\" (UID: \"1a17c173-623d-4564-9f4c-d590922b6520\") " Oct 30 05:00:21.015385 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.015362 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a17c173-623d-4564-9f4c-d590922b6520-kube-api-access-txffq\" (UniqueName: \"kubernetes.io/projected/1a17c173-623d-4564-9f4c-d590922b6520-kube-api-access-txffq\") pod \"1a17c173-623d-4564-9f4c-d590922b6520\" (UID: \"1a17c173-623d-4564-9f4c-d590922b6520\") " Oct 30 05:00:21.015934 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.015907 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a17c173-623d-4564-9f4c-d590922b6520-maas-api-service-ca" pod "1a17c173-623d-4564-9f4c-d590922b6520" (UID: "1a17c173-623d-4564-9f4c-d590922b6520"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:21.016556 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.016505 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpkcs\" (UniqueName: \"kubernetes.io/projected/f0b116c3-eb39-4ffb-b765-666e2a9a9966-kube-api-access-zpkcs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.016556 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.016535 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0225097-2d12-43e4-89b8-ea6b75c9ea54-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.016556 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.016549 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a17c173-623d-4564-9f4c-d590922b6520-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.016779 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.016566 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0b116c3-eb39-4ffb-b765-666e2a9a9966-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.016779 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.016580 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjfls\" (UniqueName: \"kubernetes.io/projected/d0225097-2d12-43e4-89b8-ea6b75c9ea54-kube-api-access-pjfls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.025489 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.025464 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a17c173-623d-4564-9f4c-d590922b6520-kube-api-access-txffq" pod "1a17c173-623d-4564-9f4c-d590922b6520" (UID: "1a17c173-623d-4564-9f4c-d590922b6520"). InnerVolumeSpecName "kube-api-access-txffq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:21.037017 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.036995 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" Oct 30 05:00:21.046018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.045994 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" Oct 30 05:00:21.117112 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.117080 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b309b4-549a-48c2-9260-fc1ae51dd264-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b309b4-549a-48c2-9260-fc1ae51dd264-maas-api-service-ca\") pod \"43b309b4-549a-48c2-9260-fc1ae51dd264\" (UID: \"43b309b4-549a-48c2-9260-fc1ae51dd264\") " Oct 30 05:00:21.118929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.117138 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3090898b-f127-44a1-85c3-ca256c7cb642-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3090898b-f127-44a1-85c3-ca256c7cb642-maas-api-service-ca\") pod \"3090898b-f127-44a1-85c3-ca256c7cb642\" (UID: \"3090898b-f127-44a1-85c3-ca256c7cb642\") " Oct 30 05:00:21.118929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.117181 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c38420-6bf8-4305-bbbf-6656194e1003-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c38420-6bf8-4305-bbbf-6656194e1003-maas-api-service-ca\") pod \"54c38420-6bf8-4305-bbbf-6656194e1003\" (UID: \"54c38420-6bf8-4305-bbbf-6656194e1003\") " Oct 30 05:00:21.118929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.117227 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3090898b-f127-44a1-85c3-ca256c7cb642-kube-api-access-c94wx\" (UniqueName: \"kubernetes.io/projected/3090898b-f127-44a1-85c3-ca256c7cb642-kube-api-access-c94wx\") pod \"3090898b-f127-44a1-85c3-ca256c7cb642\" (UID: \"3090898b-f127-44a1-85c3-ca256c7cb642\") " Oct 30 05:00:21.118929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.117487 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b309b4-549a-48c2-9260-fc1ae51dd264-kube-api-access-622hh\" (UniqueName: \"kubernetes.io/projected/43b309b4-549a-48c2-9260-fc1ae51dd264-kube-api-access-622hh\") pod \"43b309b4-549a-48c2-9260-fc1ae51dd264\" (UID: \"43b309b4-549a-48c2-9260-fc1ae51dd264\") " Oct 30 05:00:21.118929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.117572 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c38420-6bf8-4305-bbbf-6656194e1003-kube-api-access-k2ngv\" (UniqueName: \"kubernetes.io/projected/54c38420-6bf8-4305-bbbf-6656194e1003-kube-api-access-k2ngv\") pod \"54c38420-6bf8-4305-bbbf-6656194e1003\" (UID: \"54c38420-6bf8-4305-bbbf-6656194e1003\") " Oct 30 05:00:21.118929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.117740 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b309b4-549a-48c2-9260-fc1ae51dd264-maas-api-service-ca" pod "43b309b4-549a-48c2-9260-fc1ae51dd264" (UID: "43b309b4-549a-48c2-9260-fc1ae51dd264"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:21.118929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.117746 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3090898b-f127-44a1-85c3-ca256c7cb642-maas-api-service-ca" pod "3090898b-f127-44a1-85c3-ca256c7cb642" (UID: "3090898b-f127-44a1-85c3-ca256c7cb642"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:21.118929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.117856 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c38420-6bf8-4305-bbbf-6656194e1003-maas-api-service-ca" pod "54c38420-6bf8-4305-bbbf-6656194e1003" (UID: "54c38420-6bf8-4305-bbbf-6656194e1003"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:21.118929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.118696 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txffq\" (UniqueName: \"kubernetes.io/projected/1a17c173-623d-4564-9f4c-d590922b6520-kube-api-access-txffq\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.118929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.118720 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43b309b4-549a-48c2-9260-fc1ae51dd264-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.118929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.118735 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3090898b-f127-44a1-85c3-ca256c7cb642-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.118929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.118750 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54c38420-6bf8-4305-bbbf-6656194e1003-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.120215 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.120193 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3090898b-f127-44a1-85c3-ca256c7cb642-kube-api-access-c94wx" pod "3090898b-f127-44a1-85c3-ca256c7cb642" (UID: "3090898b-f127-44a1-85c3-ca256c7cb642"). InnerVolumeSpecName "kube-api-access-c94wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:21.120408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.120380 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b309b4-549a-48c2-9260-fc1ae51dd264-kube-api-access-622hh" pod "43b309b4-549a-48c2-9260-fc1ae51dd264" (UID: "43b309b4-549a-48c2-9260-fc1ae51dd264"). InnerVolumeSpecName "kube-api-access-622hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:21.120573 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.120547 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c38420-6bf8-4305-bbbf-6656194e1003-kube-api-access-k2ngv" pod "54c38420-6bf8-4305-bbbf-6656194e1003" (UID: "54c38420-6bf8-4305-bbbf-6656194e1003"). InnerVolumeSpecName "kube-api-access-k2ngv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:21.219867 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.219832 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d00032-6fea-4d6d-93d8-e481e477cc96-kube-api-access-mrsh8\" (UniqueName: \"kubernetes.io/projected/44d00032-6fea-4d6d-93d8-e481e477cc96-kube-api-access-mrsh8\") pod \"44d00032-6fea-4d6d-93d8-e481e477cc96\" (UID: \"44d00032-6fea-4d6d-93d8-e481e477cc96\") " Oct 30 05:00:21.220067 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.220051 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d00032-6fea-4d6d-93d8-e481e477cc96-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d00032-6fea-4d6d-93d8-e481e477cc96-maas-api-service-ca\") pod \"44d00032-6fea-4d6d-93d8-e481e477cc96\" (UID: \"44d00032-6fea-4d6d-93d8-e481e477cc96\") " Oct 30 05:00:21.220504 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.220472 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d00032-6fea-4d6d-93d8-e481e477cc96-maas-api-service-ca" pod "44d00032-6fea-4d6d-93d8-e481e477cc96" (UID: "44d00032-6fea-4d6d-93d8-e481e477cc96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:21.220883 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.220859 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-622hh\" (UniqueName: \"kubernetes.io/projected/43b309b4-549a-48c2-9260-fc1ae51dd264-kube-api-access-622hh\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.220975 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.220889 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2ngv\" (UniqueName: \"kubernetes.io/projected/54c38420-6bf8-4305-bbbf-6656194e1003-kube-api-access-k2ngv\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.220975 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.220905 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d00032-6fea-4d6d-93d8-e481e477cc96-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.220975 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.220919 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c94wx\" (UniqueName: \"kubernetes.io/projected/3090898b-f127-44a1-85c3-ca256c7cb642-kube-api-access-c94wx\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.221935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.221909 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d00032-6fea-4d6d-93d8-e481e477cc96-kube-api-access-mrsh8" pod "44d00032-6fea-4d6d-93d8-e481e477cc96" (UID: "44d00032-6fea-4d6d-93d8-e481e477cc96"). InnerVolumeSpecName "kube-api-access-mrsh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:21.347475 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.328287 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrsh8\" (UniqueName: \"kubernetes.io/projected/44d00032-6fea-4d6d-93d8-e481e477cc96-kube-api-access-mrsh8\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.462456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.461622 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" event={"ID":"54c38420-6bf8-4305-bbbf-6656194e1003","Type":"ContainerDied","Data":"804a4d2d1361a71c2cd332ac72616de96458017e16c0eb2ebb10d033ba8b06db"} Oct 30 05:00:21.462456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.461697 4793 scope.go:122] "RemoveContainer" containerID="f22de6650b57e86d58d7a88b73e7114b72c3f31acace7d075c98cf116b423efa" Oct 30 05:00:21.462456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.461814 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn" Oct 30 05:00:21.481688 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.480160 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" event={"ID":"f0b116c3-eb39-4ffb-b765-666e2a9a9966","Type":"ContainerDied","Data":"3c2624f28a230ce615fd36d923466bbe1e0b5127769d5e49462ccceccc74301c"} Oct 30 05:00:21.481688 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.480291 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5" Oct 30 05:00:21.503697 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.503634 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" event={"ID":"3090898b-f127-44a1-85c3-ca256c7cb642","Type":"ContainerDied","Data":"80323c6c4bd6f41e40934dabbdd3c740533c731ad6f6ae92c3ca37823f8de1a0"} Oct 30 05:00:21.503876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.503798 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b" Oct 30 05:00:21.509967 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.507652 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" event={"ID":"43b309b4-549a-48c2-9260-fc1ae51dd264","Type":"ContainerDied","Data":"28969d300e2391c9f90566897ad01e5ef282f499f1124fc13af47351dd810630"} Oct 30 05:00:21.509967 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.507751 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4" Oct 30 05:00:21.510862 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.510831 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" event={"ID":"1a17c173-623d-4564-9f4c-d590922b6520","Type":"ContainerDied","Data":"6de7ca6eac409bb1942756e2acd98901e7ee68c0bc889c3a7e6d7c0d256dad07"} Oct 30 05:00:21.513245 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.513223 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn" Oct 30 05:00:21.522816 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.522493 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" event={"ID":"d0225097-2d12-43e4-89b8-ea6b75c9ea54","Type":"ContainerDied","Data":"0e1c6df9316a922f3b82e7279562d7717638057b23c544041c0387a2557dbbf4"} Oct 30 05:00:21.522816 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.522550 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf" Oct 30 05:00:21.530227 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.528652 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" Oct 30 05:00:21.530227 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.528712 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57" event={"ID":"1f66a424-269e-475e-bf86-3e40027c9934","Type":"ContainerDied","Data":"951c5246548d786c6dc6cac6a9a3880a9e3ed40c74e9ed542051be0bcb8e12a0"} Oct 30 05:00:21.539848 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.539723 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" event={"ID":"44d00032-6fea-4d6d-93d8-e481e477cc96","Type":"ContainerDied","Data":"767cd014d6d13672ece3f7959572181688a1615c068b1c5352e4967087a86f81"} Oct 30 05:00:21.539848 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.539842 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42" Oct 30 05:00:21.546994 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.546773 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb"] Oct 30 05:00:21.579526 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.578599 4793 scope.go:122] "RemoveContainer" containerID="c1d7afe51fc348f74c4a0b9a51095fbc9167118f43e9cc85d2a757470c30615f" Oct 30 05:00:21.632859 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.632825 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn"] Oct 30 05:00:21.647178 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.647142 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-a2f984-29888940-6b7xn"] Oct 30 05:00:21.650193 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.650157 4793 scope.go:122] "RemoveContainer" containerID="653a306dee99b4c9d764752995fbc84c4eb58e9a0f4b9e9beed9d3166dff7de2" Oct 30 05:00:21.662352 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.662328 4793 scope.go:122] "RemoveContainer" containerID="88b59e505595f2bf1c3a29c025c05091021717b148404ceec5e2361d493b9896" Oct 30 05:00:21.674626 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.674602 4793 scope.go:122] "RemoveContainer" containerID="0a96021c8292b19fb81454307048fcd45316974c6d12a7b1b81af6ea2aabd732" Oct 30 05:00:21.688108 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.688086 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5"] Oct 30 05:00:21.689264 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.689238 4793 scope.go:122] "RemoveContainer" containerID="44fbee2b30b8d60d50d9028b11da3c5c66dc41c0bed4d99985fddda4df99842c" Oct 30 05:00:21.705017 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.704992 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c0e964-29888940-fhkg5"] Oct 30 05:00:21.705679 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.705659 4793 scope.go:122] "RemoveContainer" containerID="a3ec6ca0db5275e9dff2a31f8e964c28839cfa89b24226a51eb494898e6889e0" Oct 30 05:00:21.732250 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.732192 4793 scope.go:122] "RemoveContainer" containerID="0a239380d37009ea41934116bc335866b60b29054912dec127acd050de665cd8" Oct 30 05:00:21.751078 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.751017 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57"] Oct 30 05:00:21.766608 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.764984 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888940-bzb57"] Oct 30 05:00:21.780901 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.780870 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:21.797103 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.797042 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4"] Oct 30 05:00:21.810117 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.810086 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6378cc-29888940-25cx4"] Oct 30 05:00:21.850094 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.850067 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b"] Oct 30 05:00:21.858647 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.858621 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-fd27a9-29888940-jnr9b"] Oct 30 05:00:21.892117 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.892093 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" Oct 30 05:00:21.897370 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.897348 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42"] Oct 30 05:00:21.904459 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.904434 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7700a7-29888940-cpv42"] Oct 30 05:00:21.937009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.936980 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf"] Oct 30 05:00:21.947349 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.947325 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-36bbac-29888940-xfxdf"] Oct 30 05:00:21.951887 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.951867 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-kube-api-access-8zm8l\" (UniqueName: \"kubernetes.io/projected/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-kube-api-access-8zm8l\") pod \"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8\" (UID: \"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8\") " Oct 30 05:00:21.952123 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.952101 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-maas-api-service-ca\") pod \"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8\" (UID: \"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8\") " Oct 30 05:00:21.952561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.952537 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-maas-api-service-ca" pod "cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" (UID: "cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:00:21.952941 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.952920 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:21.954788 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.954755 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-kube-api-access-8zm8l" pod "cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" (UID: "cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8"). InnerVolumeSpecName "kube-api-access-8zm8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:00:21.980634 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.980614 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn"] Oct 30 05:00:21.987296 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:21.987233 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-4c2415-29888940-gdckn"] Oct 30 05:00:22.055462 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.055384 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zm8l\" (UniqueName: \"kubernetes.io/projected/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8-kube-api-access-8zm8l\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:00:22.178709 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.178674 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:22.563292 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.563266 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" Oct 30 05:00:22.563969 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.563257 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb" event={"ID":"cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8","Type":"ContainerDied","Data":"ee263736cfe5f4064632bb6abf7b0cb7ce96e564835e664303abec2ba1899d3c"} Oct 30 05:00:22.563969 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.563459 4793 scope.go:122] "RemoveContainer" containerID="cae9f6eccaae48bb5f1830a0324d57268782032a8452ad254de55dbf9aab5965" Oct 30 05:00:22.620570 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.620536 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb"] Oct 30 05:00:22.631744 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.631711 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e070ea-29888940-lg8lb"] Oct 30 05:00:22.850007 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.849935 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a17c173-623d-4564-9f4c-d590922b6520" path="/var/lib/kubelet/pods/1a17c173-623d-4564-9f4c-d590922b6520/volumes" Oct 30 05:00:22.850442 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.850428 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f66a424-269e-475e-bf86-3e40027c9934" path="/var/lib/kubelet/pods/1f66a424-269e-475e-bf86-3e40027c9934/volumes" Oct 30 05:00:22.850856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.850844 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" path="/var/lib/kubelet/pods/3090898b-f127-44a1-85c3-ca256c7cb642/volumes" Oct 30 05:00:22.851590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.851575 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" path="/var/lib/kubelet/pods/43b309b4-549a-48c2-9260-fc1ae51dd264/volumes" Oct 30 05:00:22.851990 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.851977 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" path="/var/lib/kubelet/pods/44d00032-6fea-4d6d-93d8-e481e477cc96/volumes" Oct 30 05:00:22.852361 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.852350 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" path="/var/lib/kubelet/pods/54c38420-6bf8-4305-bbbf-6656194e1003/volumes" Oct 30 05:00:22.853007 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.852990 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" path="/var/lib/kubelet/pods/cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8/volumes" Oct 30 05:00:22.853395 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.853381 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" path="/var/lib/kubelet/pods/d0225097-2d12-43e4-89b8-ea6b75c9ea54/volumes" Oct 30 05:00:22.853858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:22.853810 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" path="/var/lib/kubelet/pods/f0b116c3-eb39-4ffb-b765-666e2a9a9966/volumes" Oct 30 05:00:23.407581 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.407550 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7"] Oct 30 05:00:23.408530 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408510 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" containerName="cleanup" Oct 30 05:00:23.408592 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408531 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" containerName="cleanup" Oct 30 05:00:23.408592 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408549 4793 cpu_manager.go:414] "containerMap: removing container" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" containerName="cleanup" Oct 30 05:00:23.408592 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408554 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" containerName="cleanup" Oct 30 05:00:23.408592 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408560 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1a17c173-623d-4564-9f4c-d590922b6520" containerName="cleanup" Oct 30 05:00:23.408592 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408565 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a17c173-623d-4564-9f4c-d590922b6520" containerName="cleanup" Oct 30 05:00:23.408592 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408572 4793 cpu_manager.go:414] "containerMap: removing container" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" containerName="cleanup" Oct 30 05:00:23.408592 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408577 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" containerName="cleanup" Oct 30 05:00:23.408592 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408585 4793 cpu_manager.go:414] "containerMap: removing container" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" containerName="cleanup" Oct 30 05:00:23.408592 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408591 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408599 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408604 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408615 4793 cpu_manager.go:414] "containerMap: removing container" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408620 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408628 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408632 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408641 4793 cpu_manager.go:414] "containerMap: removing container" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408646 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408653 4793 cpu_manager.go:414] "containerMap: removing container" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408657 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408664 4793 cpu_manager.go:414] "containerMap: removing container" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408668 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408682 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1a17c173-623d-4564-9f4c-d590922b6520" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408686 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a17c173-623d-4564-9f4c-d590922b6520" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408695 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1f66a424-269e-475e-bf86-3e40027c9934" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408699 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f66a424-269e-475e-bf86-3e40027c9934" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408706 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1f66a424-269e-475e-bf86-3e40027c9934" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408710 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f66a424-269e-475e-bf86-3e40027c9934" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408719 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1a17c173-623d-4564-9f4c-d590922b6520" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408723 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a17c173-623d-4564-9f4c-d590922b6520" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408729 4793 cpu_manager.go:414] "containerMap: removing container" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408733 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408741 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408745 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408752 4793 cpu_manager.go:414] "containerMap: removing container" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408756 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408762 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1f66a424-269e-475e-bf86-3e40027c9934" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408768 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f66a424-269e-475e-bf86-3e40027c9934" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408779 4793 cpu_manager.go:414] "containerMap: removing container" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408783 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408791 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408796 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408801 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408805 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408815 4793 cpu_manager.go:414] "containerMap: removing container" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408820 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408829 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408835 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408855 4793 cpu_manager.go:414] "containerMap: removing container" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408860 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408870 4793 cpu_manager.go:414] "containerMap: removing container" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" containerName="cleanup" Oct 30 05:00:23.408925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.408874 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409002 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409011 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409019 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a17c173-623d-4564-9f4c-d590922b6520" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409024 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409029 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409036 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409042 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409047 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409055 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9c38c1-b1d7-493f-9a0d-9bba8651cbb8" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409059 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409067 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409073 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f66a424-269e-475e-bf86-3e40027c9934" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409078 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f66a424-269e-475e-bf86-3e40027c9934" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409083 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a17c173-623d-4564-9f4c-d590922b6520" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409087 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409093 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409099 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409106 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f66a424-269e-475e-bf86-3e40027c9934" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409114 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409121 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409130 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409136 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3090898b-f127-44a1-85c3-ca256c7cb642" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409144 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409149 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="291b86ef-89c7-4b35-b649-e90aa681670e" containerName="cleanup" Oct 30 05:00:23.410604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.409155 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" containerName="cleanup" Oct 30 05:00:23.415499 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.415475 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.433984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.433961 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7"] Oct 30 05:00:23.439910 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.439879 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7700a7\"/\"default-dockercfg-9r8m5\"" Oct 30 05:00:23.440048 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.439987 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7700a7\"/\"test-model-7700a7-kserve-self-signed-certs\"" Oct 30 05:00:23.454763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.454720 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7700a7\"/\"openshift-service-ca.crt\"" Oct 30 05:00:23.454877 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.454824 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7700a7\"/\"kube-root-ca.crt\"" Oct 30 05:00:23.582843 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.582807 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tls-certs\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.583282 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.582856 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-model-cache\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.583282 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.583017 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvq8s\" (UniqueName: \"kubernetes.io/projected/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-kube-api-access-mvq8s\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.583282 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.583256 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-dshm\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.583460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.583353 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-home\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.583460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.583400 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tmp-dir\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.593877 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.593842 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:23.709443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.708485 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-dshm\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.709443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.708614 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-home\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.709443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.708810 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tmp-dir\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.709443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.708898 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tls-certs\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.709443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.708937 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-model-cache\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.709443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.709018 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvq8s\" (UniqueName: \"kubernetes.io/projected/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-kube-api-access-mvq8s\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.731444 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.710227 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tmp-dir\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.731444 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.710484 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-home\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.731444 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.711296 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-model-cache\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.755084 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.755046 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tls-certs\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.765183 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.765144 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvq8s\" (UniqueName: \"kubernetes.io/projected/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-kube-api-access-mvq8s\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:23.767818 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:23.767692 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-dshm\") pod \"test-model-7700a7-kserve-6d6cb5b69f-jxgb7\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:24.030122 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:24.030035 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:24.030874 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:24.030847 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:24.348281 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:00:24.348236 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59ee8b24_fe74_4b1f_9f63_027e5fa5ec57.slice/crio-88e4882c7c5d201ab86fbf7b430927c38d4603dcd6ef2a2f67d3a814065f5bab WatchSource:0}: Error finding container 88e4882c7c5d201ab86fbf7b430927c38d4603dcd6ef2a2f67d3a814065f5bab: Status 404 returned error can't find the container with id 88e4882c7c5d201ab86fbf7b430927c38d4603dcd6ef2a2f67d3a814065f5bab Oct 30 05:00:24.348660 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:24.348636 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7"] Oct 30 05:00:24.586230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:24.586187 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" event={"ID":"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57","Type":"ContainerStarted","Data":"8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35"} Oct 30 05:00:24.586839 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:24.586239 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" event={"ID":"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57","Type":"ContainerStarted","Data":"88e4882c7c5d201ab86fbf7b430927c38d4603dcd6ef2a2f67d3a814065f5bab"} Oct 30 05:00:24.629928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:24.629834 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" podStartSLOduration=1.62981985 podStartE2EDuration="1.62981985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:24.626955949 +0000 UTC m=+3747.912830749" watchObservedRunningTime="2026-10-30 05:00:24.62981985 +0000 UTC m=+3747.915694624" Oct 30 05:00:25.031377 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:25.031312 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:25.458858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:25.458817 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:25.930917 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:25.930881 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:27.595175 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:27.595130 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:29.351687 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:29.351650 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:31.303730 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:31.303691 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:33.065088 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:33.065047 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:34.031029 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:34.030980 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:34.046517 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:34.046483 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:34.696701 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:34.696651 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:34.724183 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:34.724155 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:00:36.427944 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:36.427901 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:38.175471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:38.175362 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:39.941803 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:39.941766 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:41.220279 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:00:41.220242 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:00:41.617393 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:41.617292 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:43.480744 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:43.480704 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:45.203747 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:45.203708 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:46.802329 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:46.802282 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:47.025200 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:47.025163 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:48.724001 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:48.723958 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:50.544096 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:50.544042 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:51.896295 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.896255 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7"] Oct 30 05:00:51.897170 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897151 4793 cpu_manager.go:414] "containerMap: removing container" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" containerName="cleanup" Oct 30 05:00:51.897222 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897172 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c38420-6bf8-4305-bbbf-6656194e1003" containerName="cleanup" Oct 30 05:00:51.897222 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897185 4793 cpu_manager.go:414] "containerMap: removing container" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" containerName="cleanup" Oct 30 05:00:51.897222 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897190 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" containerName="cleanup" Oct 30 05:00:51.897222 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897199 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" containerName="cleanup" Oct 30 05:00:51.897222 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897207 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" containerName="cleanup" Oct 30 05:00:51.897384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897234 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" containerName="cleanup" Oct 30 05:00:51.897384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897239 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" containerName="cleanup" Oct 30 05:00:51.897384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897369 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d00032-6fea-4d6d-93d8-e481e477cc96" containerName="cleanup" Oct 30 05:00:51.897384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897379 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b116c3-eb39-4ffb-b765-666e2a9a9966" containerName="cleanup" Oct 30 05:00:51.897532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897388 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b309b4-549a-48c2-9260-fc1ae51dd264" containerName="cleanup" Oct 30 05:00:51.897532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897396 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a17c173-623d-4564-9f4c-d590922b6520" containerName="cleanup" Oct 30 05:00:51.897532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.897404 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0225097-2d12-43e4-89b8-ea6b75c9ea54" containerName="cleanup" Oct 30 05:00:51.901767 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.901750 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:51.918807 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.918785 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-82bec5-openshift-default-dockercfg-sz8f7\"" Oct 30 05:00:51.937785 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:51.937753 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7"] Oct 30 05:00:52.029554 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.029525 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gj9r\" (UniqueName: \"kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-kube-api-access-8gj9r\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.029731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.029569 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-socket\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.029731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.029691 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-certs\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.029869 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.029751 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-credential-socket\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.029869 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.029809 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a966611-bdd4-4210-8908-a1e3eee59590-istiod-ca-cert\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.029984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.029917 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a966611-bdd4-4210-8908-a1e3eee59590-istio-podinfo\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.030044 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.030008 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-envoy\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.030117 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.030101 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-istio-token\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.030258 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.030243 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-data\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.132562 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.132529 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-certs\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.132821 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.132583 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-credential-socket\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.132821 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.132630 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a966611-bdd4-4210-8908-a1e3eee59590-istiod-ca-cert\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.132821 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.132690 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a966611-bdd4-4210-8908-a1e3eee59590-istio-podinfo\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.132821 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.132807 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-envoy\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.133148 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.132888 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-istio-token\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.133148 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.133052 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-certs\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.133148 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.133083 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-credential-socket\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.133148 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.133087 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-data\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.133381 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.133181 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gj9r\" (UniqueName: \"kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-kube-api-access-8gj9r\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.133381 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.133226 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-socket\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.133520 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.133385 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a966611-bdd4-4210-8908-a1e3eee59590-istiod-ca-cert\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.133627 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.133601 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-socket\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.133734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.133698 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-data\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.135577 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.135556 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a966611-bdd4-4210-8908-a1e3eee59590-istio-podinfo\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.135705 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.135544 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-envoy\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.156091 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.156028 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-istio-token\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.156232 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.156213 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gj9r\" (UniqueName: \"kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-kube-api-access-8gj9r\") pod \"e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.217763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.217736 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:52.300256 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.300221 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:52.604720 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:00:52.604688 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a966611_bdd4_4210_8908_a1e3eee59590.slice/crio-f6b2dc32e2202816485d299f036c071261373d90edf50d1e710690745b195dd2 WatchSource:0}: Error finding container f6b2dc32e2202816485d299f036c071261373d90edf50d1e710690745b195dd2: Status 404 returned error can't find the container with id f6b2dc32e2202816485d299f036c071261373d90edf50d1e710690745b195dd2 Oct 30 05:00:52.605366 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.605340 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7"] Oct 30 05:00:52.607709 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.607675 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:00:52.607805 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.607736 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:00:52.607805 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.607765 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:00:52.892818 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.892723 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" event={"ID":"0a966611-bdd4-4210-8908-a1e3eee59590","Type":"ContainerStarted","Data":"dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432"} Oct 30 05:00:52.892818 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.892787 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" event={"ID":"0a966611-bdd4-4210-8908-a1e3eee59590","Type":"ContainerStarted","Data":"f6b2dc32e2202816485d299f036c071261373d90edf50d1e710690745b195dd2"} Oct 30 05:00:52.940497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:52.940449 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" podStartSLOduration=1.940434856 podStartE2EDuration="1.940434856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:00:52.936488998 +0000 UTC m=+3776.222363773" watchObservedRunningTime="2026-10-30 05:00:52.940434856 +0000 UTC m=+3776.226309675" Oct 30 05:00:53.218155 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:53.218126 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:00:53.219349 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:53.219330 4793 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 05:00:53.219457 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:53.219372 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" podUID="0a966611-bdd4-4210-8908-a1e3eee59590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 05:00:54.104047 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:54.104009 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:54.218212 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:54.218180 4793 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 05:00:54.218405 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:54.218230 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" podUID="0a966611-bdd4-4210-8908-a1e3eee59590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 05:00:55.218775 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:55.218739 4793 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 05:00:55.219151 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:55.218794 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" podUID="0a966611-bdd4-4210-8908-a1e3eee59590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 05:00:55.949711 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:55.949674 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:56.218781 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:56.218695 4793 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 05:00:56.218781 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:56.218762 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" podUID="0a966611-bdd4-4210-8908-a1e3eee59590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 05:00:57.219006 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:57.218965 4793 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 05:00:57.219540 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:57.219020 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" podUID="0a966611-bdd4-4210-8908-a1e3eee59590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 05:00:57.695018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:57.694965 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:57.922142 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:57.922102 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:00:58.226109 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:58.226068 4793 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 05:00:58.227536 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:58.227494 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" podUID="0a966611-bdd4-4210-8908-a1e3eee59590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 05:00:59.218956 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:59.218921 4793 patch_prober.go:29] interesting pod/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 05:00:59.219132 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:59.218975 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" podUID="0a966611-bdd4-4210-8908-a1e3eee59590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 05:00:59.505622 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:00:59.505574 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:00.224081 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:00.224052 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:01:00.224400 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:00.224367 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:01:00.225190 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:00.225164 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:01:01.193982 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:01.193947 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:03.113565 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:03.113528 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:04.798477 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:04.798430 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:06.555801 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:06.555764 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:08.335754 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:08.335714 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:08.583200 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:08.583168 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:08.804902 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:08.804864 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:10.339176 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:10.339139 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:10.841439 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:10.838017 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:11.220771 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:01:11.220738 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:01:12.105695 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:12.105655 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:13.910553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:13.910509 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:15.604928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:15.604892 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:17.244474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:17.244444 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:18.929561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:18.929507 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:20.874699 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:20.874657 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:22.633408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:22.633373 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:24.306768 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:24.306728 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:26.038868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:26.038828 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:27.851518 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:27.851477 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:29.219447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:29.219040 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:29.848565 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:29.848527 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:31.688479 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:31.688440 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:31.985632 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:31.985544 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:33.944224 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:33.944190 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:34.207597 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:34.204147 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:34.623591 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:34.623499 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:35.140552 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:35.140519 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:35.368327 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:35.368267 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:35.853770 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:35.853727 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:35.971228 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:35.971190 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc"] Oct 30 05:01:35.971459 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:35.971405 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" containerID="cri-o://442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02" gracePeriod=30 Oct 30 05:01:37.365442 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:37.364577 4793 patch_prober.go:29] interesting pod/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:01:37.365442 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:37.364655 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:01:37.871489 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:37.861386 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:40.085543 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:40.085447 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:41.227041 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:01:41.226947 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:01:41.273514 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.273489 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 05:01:41.347544 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.347515 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-data\") pod \"b3abc285-621f-4873-a77d-70d6f93a6c53\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " Oct 30 05:01:41.347739 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.347573 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3abc285-621f-4873-a77d-70d6f93a6c53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3abc285-621f-4873-a77d-70d6f93a6c53-istiod-ca-cert\") pod \"b3abc285-621f-4873-a77d-70d6f93a6c53\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " Oct 30 05:01:41.347739 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.347664 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-socket\") pod \"b3abc285-621f-4873-a77d-70d6f93a6c53\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " Oct 30 05:01:41.347739 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.347708 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-credential-socket\") pod \"b3abc285-621f-4873-a77d-70d6f93a6c53\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " Oct 30 05:01:41.347915 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.347845 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-data" pod "b3abc285-621f-4873-a77d-70d6f93a6c53" (UID: "b3abc285-621f-4873-a77d-70d6f93a6c53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:01:41.347915 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.347851 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-kube-api-access-lw5v6\" (UniqueName: \"kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-kube-api-access-lw5v6\") pod \"b3abc285-621f-4873-a77d-70d6f93a6c53\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " Oct 30 05:01:41.348041 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.347982 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-token\" (UniqueName: \"kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-token\") pod \"b3abc285-621f-4873-a77d-70d6f93a6c53\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " Oct 30 05:01:41.348102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.348054 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-podinfo\") pod \"b3abc285-621f-4873-a77d-70d6f93a6c53\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " Oct 30 05:01:41.348102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.348086 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3abc285-621f-4873-a77d-70d6f93a6c53-istiod-ca-cert" pod "b3abc285-621f-4873-a77d-70d6f93a6c53" (UID: "b3abc285-621f-4873-a77d-70d6f93a6c53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:01:41.348222 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.348100 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-envoy\") pod \"b3abc285-621f-4873-a77d-70d6f93a6c53\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " Oct 30 05:01:41.348222 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.348152 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-certs\") pod \"b3abc285-621f-4873-a77d-70d6f93a6c53\" (UID: \"b3abc285-621f-4873-a77d-70d6f93a6c53\") " Oct 30 05:01:41.348521 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.348386 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-socket" pod "b3abc285-621f-4873-a77d-70d6f93a6c53" (UID: "b3abc285-621f-4873-a77d-70d6f93a6c53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:01:41.348668 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.348558 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-credential-socket" pod "b3abc285-621f-4873-a77d-70d6f93a6c53" (UID: "b3abc285-621f-4873-a77d-70d6f93a6c53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:01:41.349185 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.349104 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:41.349185 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.349132 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3abc285-621f-4873-a77d-70d6f93a6c53-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:41.349185 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.349147 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:41.349185 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.349160 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:41.349494 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.349218 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-certs" pod "b3abc285-621f-4873-a77d-70d6f93a6c53" (UID: "b3abc285-621f-4873-a77d-70d6f93a6c53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:01:41.351868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.351832 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-envoy" pod "b3abc285-621f-4873-a77d-70d6f93a6c53" (UID: "b3abc285-621f-4873-a77d-70d6f93a6c53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:01:41.351978 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.351948 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-token" pod "b3abc285-621f-4873-a77d-70d6f93a6c53" (UID: "b3abc285-621f-4873-a77d-70d6f93a6c53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:01:41.352214 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.352160 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-podinfo" pod "b3abc285-621f-4873-a77d-70d6f93a6c53" (UID: "b3abc285-621f-4873-a77d-70d6f93a6c53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:01:41.355186 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.355143 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-kube-api-access-lw5v6" pod "b3abc285-621f-4873-a77d-70d6f93a6c53" (UID: "b3abc285-621f-4873-a77d-70d6f93a6c53"). InnerVolumeSpecName "kube-api-access-lw5v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:01:41.451031 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.450981 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw5v6\" (UniqueName: \"kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-kube-api-access-lw5v6\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:41.451031 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.451031 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:41.451031 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.451041 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:41.451289 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.451050 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:41.451289 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.451060 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3abc285-621f-4873-a77d-70d6f93a6c53-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:41.502684 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.502605 4793 generic.go:356] "Generic (PLEG): container finished" podID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerID="442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02" exitCode=0 Oct 30 05:01:41.502854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.502685 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" Oct 30 05:01:41.502854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.502695 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" event={"ID":"b3abc285-621f-4873-a77d-70d6f93a6c53","Type":"ContainerDied","Data":"442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02"} Oct 30 05:01:41.502854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.502761 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc" event={"ID":"b3abc285-621f-4873-a77d-70d6f93a6c53","Type":"ContainerDied","Data":"16153143acfd253f3b26f85313ae9c0e14b8c4bcd6ffa35f6cac785695178889"} Oct 30 05:01:41.502854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.502793 4793 scope.go:122] "RemoveContainer" containerID="442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02" Oct 30 05:01:41.518142 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.518126 4793 scope.go:122] "RemoveContainer" containerID="3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950" Oct 30 05:01:41.529509 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.529486 4793 scope.go:122] "RemoveContainer" containerID="442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02" Oct 30 05:01:41.529771 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:01:41.529748 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02\": container with ID starting with 442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02 not found: ID does not exist" containerID="442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02" Oct 30 05:01:41.529827 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.529783 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02"} err="failed to get container status \"442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02\": rpc error: code = NotFound desc = could not find container \"442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02\": container with ID starting with 442572820ff7c5ca182d1ccd5908e8ff4eaaedbca77888f961238d4b9cf50c02 not found: ID does not exist" Oct 30 05:01:41.529827 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.529800 4793 scope.go:122] "RemoveContainer" containerID="3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950" Oct 30 05:01:41.530034 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:01:41.530018 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950\": container with ID starting with 3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950 not found: ID does not exist" containerID="3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950" Oct 30 05:01:41.530077 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.530038 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950"} err="failed to get container status \"3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950\": rpc error: code = NotFound desc = could not find container \"3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950\": container with ID starting with 3d5e5d79493a499ecc3890753b17c7128ad8b2ecf75a6f3dc080f625b230e950 not found: ID does not exist" Oct 30 05:01:41.559255 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.559230 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc"] Oct 30 05:01:41.565222 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:41.565201 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-91705f-openshift-default-7659df5f6f-wx9kc"] Oct 30 05:01:42.080634 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:42.080586 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:42.848539 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:42.848509 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" path="/var/lib/kubelet/pods/b3abc285-621f-4873-a77d-70d6f93a6c53/volumes" Oct 30 05:01:43.787920 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:43.787799 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:44.001717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:44.001681 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:44.254892 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:44.254856 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:44.651677 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:44.651595 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:44.949289 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:44.949248 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:45.174443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:45.174200 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn"] Oct 30 05:01:45.174980 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:45.174456 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" podUID="f256eec9-b172-444a-b3fe-a311bf865e01" containerName="istio-proxy" containerID="cri-o://f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8" gracePeriod=30 Oct 30 05:01:45.969885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:45.969850 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:01:45.970059 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:45.969899 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" podUID="f256eec9-b172-444a-b3fe-a311bf865e01" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:01:46.221086 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:46.220997 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:47.900329 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:47.900294 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:49.694986 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:49.694950 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:50.188733 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.188699 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:50.424406 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.424344 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:50.488681 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.488655 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 05:01:50.541286 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.541261 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f256eec9-b172-444a-b3fe-a311bf865e01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f256eec9-b172-444a-b3fe-a311bf865e01-istiod-ca-cert\") pod \"f256eec9-b172-444a-b3fe-a311bf865e01\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " Oct 30 05:01:50.541478 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.541359 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-credential-socket\") pod \"f256eec9-b172-444a-b3fe-a311bf865e01\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " Oct 30 05:01:50.541478 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.541461 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f256eec9-b172-444a-b3fe-a311bf865e01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f256eec9-b172-444a-b3fe-a311bf865e01-istio-podinfo\") pod \"f256eec9-b172-444a-b3fe-a311bf865e01\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " Oct 30 05:01:50.541626 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.541489 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-socket\") pod \"f256eec9-b172-444a-b3fe-a311bf865e01\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " Oct 30 05:01:50.541626 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.541564 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-kube-api-access-stqz4\" (UniqueName: \"kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-kube-api-access-stqz4\") pod \"f256eec9-b172-444a-b3fe-a311bf865e01\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " Oct 30 05:01:50.541626 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.541595 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-envoy\") pod \"f256eec9-b172-444a-b3fe-a311bf865e01\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " Oct 30 05:01:50.541787 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.541684 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-istio-token\" (UniqueName: \"kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-istio-token\") pod \"f256eec9-b172-444a-b3fe-a311bf865e01\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " Oct 30 05:01:50.541787 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.541692 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f256eec9-b172-444a-b3fe-a311bf865e01-istiod-ca-cert" pod "f256eec9-b172-444a-b3fe-a311bf865e01" (UID: "f256eec9-b172-444a-b3fe-a311bf865e01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:01:50.541787 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.541696 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-credential-socket" pod "f256eec9-b172-444a-b3fe-a311bf865e01" (UID: "f256eec9-b172-444a-b3fe-a311bf865e01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:01:50.541942 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.541805 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-socket" pod "f256eec9-b172-444a-b3fe-a311bf865e01" (UID: "f256eec9-b172-444a-b3fe-a311bf865e01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:01:50.541942 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.541844 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-data\") pod \"f256eec9-b172-444a-b3fe-a311bf865e01\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " Oct 30 05:01:50.542054 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.542004 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-certs\") pod \"f256eec9-b172-444a-b3fe-a311bf865e01\" (UID: \"f256eec9-b172-444a-b3fe-a311bf865e01\") " Oct 30 05:01:50.542352 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.542286 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-data" pod "f256eec9-b172-444a-b3fe-a311bf865e01" (UID: "f256eec9-b172-444a-b3fe-a311bf865e01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:01:50.542552 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.542533 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-certs" pod "f256eec9-b172-444a-b3fe-a311bf865e01" (UID: "f256eec9-b172-444a-b3fe-a311bf865e01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:01:50.542935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.542907 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:50.542935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.542936 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f256eec9-b172-444a-b3fe-a311bf865e01-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:50.543113 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.542946 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:50.543113 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.542955 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:50.543113 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.542964 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:50.544337 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.544308 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f256eec9-b172-444a-b3fe-a311bf865e01-istio-podinfo" pod "f256eec9-b172-444a-b3fe-a311bf865e01" (UID: "f256eec9-b172-444a-b3fe-a311bf865e01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:01:50.544467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.544406 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-kube-api-access-stqz4" pod "f256eec9-b172-444a-b3fe-a311bf865e01" (UID: "f256eec9-b172-444a-b3fe-a311bf865e01"). InnerVolumeSpecName "kube-api-access-stqz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:01:50.544546 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.544493 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-envoy" pod "f256eec9-b172-444a-b3fe-a311bf865e01" (UID: "f256eec9-b172-444a-b3fe-a311bf865e01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:01:50.544602 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.544568 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-istio-token" pod "f256eec9-b172-444a-b3fe-a311bf865e01" (UID: "f256eec9-b172-444a-b3fe-a311bf865e01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:01:50.610088 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.610047 4793 generic.go:356] "Generic (PLEG): container finished" podID="f256eec9-b172-444a-b3fe-a311bf865e01" containerID="f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8" exitCode=0 Oct 30 05:01:50.610247 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.610093 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" Oct 30 05:01:50.610247 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.610128 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" event={"ID":"f256eec9-b172-444a-b3fe-a311bf865e01","Type":"ContainerDied","Data":"f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8"} Oct 30 05:01:50.610247 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.610196 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn" event={"ID":"f256eec9-b172-444a-b3fe-a311bf865e01","Type":"ContainerDied","Data":"174595ba57459ecbc8fcddd200932e1c4870b91cbc2d803e09a07e34ed4acaac"} Oct 30 05:01:50.610247 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.610228 4793 scope.go:122] "RemoveContainer" containerID="f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8" Oct 30 05:01:50.623747 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.623726 4793 scope.go:122] "RemoveContainer" containerID="f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8" Oct 30 05:01:50.624080 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:01:50.624057 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8\": container with ID starting with f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8 not found: ID does not exist" containerID="f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8" Oct 30 05:01:50.624173 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.624086 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8"} err="failed to get container status \"f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8\": rpc error: code = NotFound desc = could not find container \"f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8\": container with ID starting with f3c670df3639ee6051936f0d1ad4b1a3139d436d4fe33128791afa885f2bfdf8 not found: ID does not exist" Oct 30 05:01:50.644044 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.644022 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f256eec9-b172-444a-b3fe-a311bf865e01-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:50.644122 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.644045 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stqz4\" (UniqueName: \"kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-kube-api-access-stqz4\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:50.644122 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.644057 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f256eec9-b172-444a-b3fe-a311bf865e01-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:50.644122 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.644065 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f256eec9-b172-444a-b3fe-a311bf865e01-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:01:50.679538 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.679505 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn"] Oct 30 05:01:50.684216 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.684188 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-mb6fn"] Oct 30 05:01:50.850572 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:50.850501 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f256eec9-b172-444a-b3fe-a311bf865e01" path="/var/lib/kubelet/pods/f256eec9-b172-444a-b3fe-a311bf865e01/volumes" Oct 30 05:01:51.435838 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:51.435802 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:53.063280 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:53.063243 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:54.836848 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:54.836795 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:56.593170 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:56.592969 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:57.841642 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:57.841606 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:01:58.307144 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:01:58.307098 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:00.127724 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:00.127685 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:01.245320 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:01.245284 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:01.727072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:01.727029 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:03.443111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:03.443074 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:05.193018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:05.192977 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:06.828985 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:06.828938 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:08.286401 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:08.286343 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:09.035088 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:09.035051 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:10.654471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:10.654387 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:11.220623 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:02:11.220586 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:02:12.379598 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:12.379554 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:14.122866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:14.122821 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:15.432940 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:15.432901 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:16.638456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:16.635533 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq"] Oct 30 05:02:16.638456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:16.635806 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" containerID="cri-o://ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03" gracePeriod=30 Oct 30 05:02:17.383123 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:17.383078 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:17.602936 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:17.602905 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:17.915976 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:17.915935 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:19.520239 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:19.520201 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:21.147203 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:21.147166 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:21.958864 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:21.958835 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 05:02:22.039798 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.039762 4793 generic.go:356] "Generic (PLEG): container finished" podID="ac37ac34-46aa-4ad9-889b-117548e46735" containerID="ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03" exitCode=0 Oct 30 05:02:22.039971 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.039822 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" Oct 30 05:02:22.039971 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.039844 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" event={"ID":"ac37ac34-46aa-4ad9-889b-117548e46735","Type":"ContainerDied","Data":"ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03"} Oct 30 05:02:22.039971 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.039924 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq" event={"ID":"ac37ac34-46aa-4ad9-889b-117548e46735","Type":"ContainerDied","Data":"eb24019992deba55b04e1ff9200d2c5bb95bee33954309f40175746808061e5e"} Oct 30 05:02:22.039971 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.039953 4793 scope.go:122] "RemoveContainer" containerID="ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03" Oct 30 05:02:22.051513 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.051489 4793 scope.go:122] "RemoveContainer" containerID="a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13" Oct 30 05:02:22.056621 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.056598 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-kube-api-access-r9h9n\" (UniqueName: \"kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-kube-api-access-r9h9n\") pod \"ac37ac34-46aa-4ad9-889b-117548e46735\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " Oct 30 05:02:22.057050 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.057026 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-socket\") pod \"ac37ac34-46aa-4ad9-889b-117548e46735\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " Oct 30 05:02:22.057181 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.057163 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-envoy\") pod \"ac37ac34-46aa-4ad9-889b-117548e46735\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " Oct 30 05:02:22.057247 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.057220 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac37ac34-46aa-4ad9-889b-117548e46735-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac37ac34-46aa-4ad9-889b-117548e46735-istio-podinfo\") pod \"ac37ac34-46aa-4ad9-889b-117548e46735\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " Oct 30 05:02:22.057307 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.057271 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-credential-socket\") pod \"ac37ac34-46aa-4ad9-889b-117548e46735\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " Oct 30 05:02:22.057364 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.057305 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-data\") pod \"ac37ac34-46aa-4ad9-889b-117548e46735\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " Oct 30 05:02:22.057364 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.057349 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-istio-token\" (UniqueName: \"kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-istio-token\") pod \"ac37ac34-46aa-4ad9-889b-117548e46735\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " Oct 30 05:02:22.057525 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.057441 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-certs\") pod \"ac37ac34-46aa-4ad9-889b-117548e46735\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " Oct 30 05:02:22.057585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.057571 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac37ac34-46aa-4ad9-889b-117548e46735-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac37ac34-46aa-4ad9-889b-117548e46735-istiod-ca-cert\") pod \"ac37ac34-46aa-4ad9-889b-117548e46735\" (UID: \"ac37ac34-46aa-4ad9-889b-117548e46735\") " Oct 30 05:02:22.057870 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.057845 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-data" pod "ac37ac34-46aa-4ad9-889b-117548e46735" (UID: "ac37ac34-46aa-4ad9-889b-117548e46735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:22.058030 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.058006 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-socket" pod "ac37ac34-46aa-4ad9-889b-117548e46735" (UID: "ac37ac34-46aa-4ad9-889b-117548e46735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:22.058234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.058212 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-credential-socket" pod "ac37ac34-46aa-4ad9-889b-117548e46735" (UID: "ac37ac34-46aa-4ad9-889b-117548e46735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:22.060876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.058841 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:22.060876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.058867 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:22.060876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.058880 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:22.060876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.059167 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-kube-api-access-r9h9n" pod "ac37ac34-46aa-4ad9-889b-117548e46735" (UID: "ac37ac34-46aa-4ad9-889b-117548e46735"). InnerVolumeSpecName "kube-api-access-r9h9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:22.060876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.059578 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac37ac34-46aa-4ad9-889b-117548e46735-istiod-ca-cert" pod "ac37ac34-46aa-4ad9-889b-117548e46735" (UID: "ac37ac34-46aa-4ad9-889b-117548e46735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:22.060876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.059807 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-certs" pod "ac37ac34-46aa-4ad9-889b-117548e46735" (UID: "ac37ac34-46aa-4ad9-889b-117548e46735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:22.070834 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.061259 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-istio-token" pod "ac37ac34-46aa-4ad9-889b-117548e46735" (UID: "ac37ac34-46aa-4ad9-889b-117548e46735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:22.070834 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.062514 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-envoy" pod "ac37ac34-46aa-4ad9-889b-117548e46735" (UID: "ac37ac34-46aa-4ad9-889b-117548e46735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:22.070834 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.068670 4793 scope.go:122] "RemoveContainer" containerID="ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03" Oct 30 05:02:22.070834 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:02:22.069026 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03\": container with ID starting with ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03 not found: ID does not exist" containerID="ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03" Oct 30 05:02:22.070834 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.069064 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03"} err="failed to get container status \"ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03\": rpc error: code = NotFound desc = could not find container \"ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03\": container with ID starting with ae99db4b7f79cf8ecddbfd9304afca89986f4a330a278a1f26e438f7d98beb03 not found: ID does not exist" Oct 30 05:02:22.070834 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.069088 4793 scope.go:122] "RemoveContainer" containerID="a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13" Oct 30 05:02:22.070834 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:02:22.069360 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13\": container with ID starting with a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13 not found: ID does not exist" containerID="a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13" Oct 30 05:02:22.070834 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.069387 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13"} err="failed to get container status \"a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13\": rpc error: code = NotFound desc = could not find container \"a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13\": container with ID starting with a9ceec3009d54721ceb9a36f62180e925812f769e1a67619e330fe3d16d75d13 not found: ID does not exist" Oct 30 05:02:22.073072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.073051 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac37ac34-46aa-4ad9-889b-117548e46735-istio-podinfo" pod "ac37ac34-46aa-4ad9-889b-117548e46735" (UID: "ac37ac34-46aa-4ad9-889b-117548e46735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:02:22.160245 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.160204 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac37ac34-46aa-4ad9-889b-117548e46735-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:22.160245 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.160233 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9h9n\" (UniqueName: \"kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-kube-api-access-r9h9n\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:22.160767 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.160248 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:22.160767 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.160265 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac37ac34-46aa-4ad9-889b-117548e46735-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:22.160767 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.160278 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac37ac34-46aa-4ad9-889b-117548e46735-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:22.160767 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.160291 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac37ac34-46aa-4ad9-889b-117548e46735-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:22.388531 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.388501 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq"] Oct 30 05:02:22.403281 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.403251 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-134dad-openshift-default-7d94b74d4f-psjdq"] Oct 30 05:02:22.849256 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.849221 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" path="/var/lib/kubelet/pods/ac37ac34-46aa-4ad9-889b-117548e46735/volumes" Oct 30 05:02:22.855027 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:22.854993 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:23.816370 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:23.816335 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m"] Oct 30 05:02:24.080941 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:24.080831 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:24.293889 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:24.293859 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:24.735751 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:24.735714 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:26.370841 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:26.370803 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:28.023165 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:28.023129 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:28.246814 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:28.246778 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:28.817648 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:28.817609 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" podUID="c658db96-88c7-42b3-9cf3-5fcc5c45a21e" containerName="payload-processing" containerID="cri-o://ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7" gracePeriod=25 Oct 30 05:02:29.714533 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:29.714500 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:29.874053 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:29.874017 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" podUID="c658db96-88c7-42b3-9cf3-5fcc5c45a21e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:02:30.522956 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:30.522918 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:31.504246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:31.504210 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:33.216956 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:33.216912 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:34.964174 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:34.964130 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:36.775187 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:36.775152 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:37.490567 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:37.490533 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729"] Oct 30 05:02:37.490753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:37.490731 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" containerID="cri-o://f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad" gracePeriod=30 Oct 30 05:02:38.140976 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.140943 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7"] Oct 30 05:02:38.141774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.141738 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" podUID="6b6e51a1-7f4e-492f-92b1-b0025a43fef0" containerName="maas-api" containerID="cri-o://e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3" gracePeriod=30 Oct 30 05:02:38.314772 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.314736 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:38.538783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.538755 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 05:02:38.750497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.730107 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-maas-api-tls\") pod \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " Oct 30 05:02:38.750497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.730170 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-metrics-certs\") pod \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " Oct 30 05:02:38.750497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.730241 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-kube-api-access-j9229\" (UniqueName: \"kubernetes.io/projected/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-kube-api-access-j9229\") pod \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " Oct 30 05:02:38.750497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.730459 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-openshift-service-ca\") pod \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\" (UID: \"6b6e51a1-7f4e-492f-92b1-b0025a43fef0\") " Oct 30 05:02:38.758817 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.758780 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-openshift-service-ca" pod "6b6e51a1-7f4e-492f-92b1-b0025a43fef0" (UID: "6b6e51a1-7f4e-492f-92b1-b0025a43fef0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:38.798736 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.797510 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-kube-api-access-j9229" pod "6b6e51a1-7f4e-492f-92b1-b0025a43fef0" (UID: "6b6e51a1-7f4e-492f-92b1-b0025a43fef0"). InnerVolumeSpecName "kube-api-access-j9229". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:38.800557 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.800522 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-metrics-certs" pod "6b6e51a1-7f4e-492f-92b1-b0025a43fef0" (UID: "6b6e51a1-7f4e-492f-92b1-b0025a43fef0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:38.800722 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.800547 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-maas-api-tls" pod "6b6e51a1-7f4e-492f-92b1-b0025a43fef0" (UID: "6b6e51a1-7f4e-492f-92b1-b0025a43fef0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:38.832708 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.832682 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:38.832874 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.832714 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:38.832874 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.832728 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9229\" (UniqueName: \"kubernetes.io/projected/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-kube-api-access-j9229\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:38.832874 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:38.832741 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6e51a1-7f4e-492f-92b1-b0025a43fef0-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:39.084841 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.084754 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:39.273441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.266782 4793 generic.go:356] "Generic (PLEG): container finished" podID="6b6e51a1-7f4e-492f-92b1-b0025a43fef0" containerID="e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3" exitCode=0 Oct 30 05:02:39.273441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.266921 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" event={"ID":"6b6e51a1-7f4e-492f-92b1-b0025a43fef0","Type":"ContainerDied","Data":"e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3"} Oct 30 05:02:39.273441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.266974 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" event={"ID":"6b6e51a1-7f4e-492f-92b1-b0025a43fef0","Type":"ContainerDied","Data":"ef154586dc9ff5a4b9366a8cdec24d37496bab26d172d8f72b3fbdc8825f2a7d"} Oct 30 05:02:39.273441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.267004 4793 scope.go:122] "RemoveContainer" containerID="e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3" Oct 30 05:02:39.273441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.267166 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7" Oct 30 05:02:39.286880 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.286850 4793 scope.go:122] "RemoveContainer" containerID="e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3" Oct 30 05:02:39.287229 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:02:39.287199 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3\": container with ID starting with e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3 not found: ID does not exist" containerID="e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3" Oct 30 05:02:39.287347 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.287239 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3"} err="failed to get container status \"e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3\": rpc error: code = NotFound desc = could not find container \"e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3\": container with ID starting with e6e0a7df99b680ed172598a7c5bd4f15093b913fa2b997ca55a85d363f3a45d3 not found: ID does not exist" Oct 30 05:02:39.303172 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.303139 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:39.327348 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.327325 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7"] Oct 30 05:02:39.339601 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.339539 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-5f1508-6fdbd7c78c-xjld7"] Oct 30 05:02:39.784928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.784888 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:39.874638 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:39.874604 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" podUID="c658db96-88c7-42b3-9cf3-5fcc5c45a21e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:02:40.157698 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:40.157607 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:40.848353 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:40.848322 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6e51a1-7f4e-492f-92b1-b0025a43fef0" path="/var/lib/kubelet/pods/6b6e51a1-7f4e-492f-92b1-b0025a43fef0/volumes" Oct 30 05:02:40.944811 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:40.944770 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:41.180041 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:41.179997 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:41.220889 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:02:41.220851 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:02:41.608728 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:41.608628 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts"] Oct 30 05:02:41.608916 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:41.608827 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" containerID="cri-o://df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f" gracePeriod=30 Oct 30 05:02:41.967226 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:41.967194 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:02:41.967681 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:41.967251 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:02:42.683261 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.683225 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:42.804795 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.804766 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 05:02:42.913114 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.913072 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-certs\") pod \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " Oct 30 05:02:42.913307 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.913206 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-envoy\") pod \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " Oct 30 05:02:42.913383 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.913331 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-podinfo\") pod \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " Oct 30 05:02:42.913471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.913376 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-certs" pod "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" (UID: "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:42.913471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.913410 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-credential-socket\") pod \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " Oct 30 05:02:42.913596 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.913546 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-kube-api-access-ds6cc\" (UniqueName: \"kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-kube-api-access-ds6cc\") pod \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " Oct 30 05:02:42.913596 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.913581 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istiod-ca-cert\") pod \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " Oct 30 05:02:42.913703 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.913686 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-data\") pod \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " Oct 30 05:02:42.913773 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.913705 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-credential-socket" pod "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" (UID: "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:42.913773 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.913730 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-token\" (UniqueName: \"kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-token\") pod \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " Oct 30 05:02:42.913773 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.913763 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-socket\") pod \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\" (UID: \"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc\") " Oct 30 05:02:42.914012 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.913994 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istiod-ca-cert" pod "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" (UID: "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:42.914825 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.914768 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-data" pod "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" (UID: "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:42.915060 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.914858 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:42.915060 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.914897 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:42.915060 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.914913 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:42.915540 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.915516 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-socket" pod "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" (UID: "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:42.916453 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.916394 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-podinfo" pod "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" (UID: "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:02:42.917095 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.917067 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-kube-api-access-ds6cc" pod "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" (UID: "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc"). InnerVolumeSpecName "kube-api-access-ds6cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:42.917376 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.917351 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-envoy" pod "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" (UID: "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:42.918194 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:42.918170 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-token" pod "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" (UID: "b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:43.016116 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.016033 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:43.016116 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.016066 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:43.016116 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.016080 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds6cc\" (UniqueName: \"kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-kube-api-access-ds6cc\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:43.016116 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.016098 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:43.016116 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.016113 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:43.016625 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.016126 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:43.371878 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.371784 4793 generic.go:356] "Generic (PLEG): container finished" podID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerID="f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad" exitCode=0 Oct 30 05:02:43.371878 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.371831 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" Oct 30 05:02:43.372086 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.371868 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" event={"ID":"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc","Type":"ContainerDied","Data":"f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad"} Oct 30 05:02:43.372086 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.371927 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729" event={"ID":"b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc","Type":"ContainerDied","Data":"e3f2b648608f896f9bf4e2812084b8992153d12f509411ee8d19e25f6ef4af8c"} Oct 30 05:02:43.372086 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.371956 4793 scope.go:122] "RemoveContainer" containerID="f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad" Oct 30 05:02:43.385470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.385450 4793 scope.go:122] "RemoveContainer" containerID="88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639" Oct 30 05:02:43.399692 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.399671 4793 scope.go:122] "RemoveContainer" containerID="f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad" Oct 30 05:02:43.400006 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:02:43.399982 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad\": container with ID starting with f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad not found: ID does not exist" containerID="f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad" Oct 30 05:02:43.400082 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.400017 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad"} err="failed to get container status \"f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad\": rpc error: code = NotFound desc = could not find container \"f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad\": container with ID starting with f27bba02dc952b836e416881f4e02e7e8b05db3fbbd35780222d975c571a10ad not found: ID does not exist" Oct 30 05:02:43.400082 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.400041 4793 scope.go:122] "RemoveContainer" containerID="88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639" Oct 30 05:02:43.400317 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:02:43.400296 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639\": container with ID starting with 88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639 not found: ID does not exist" containerID="88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639" Oct 30 05:02:43.400368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.400321 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639"} err="failed to get container status \"88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639\": rpc error: code = NotFound desc = could not find container \"88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639\": container with ID starting with 88bcb4b5aabbf45820124ba2d7c65432175bf197302ed28d241088f6b8c8a639 not found: ID does not exist" Oct 30 05:02:43.428095 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.428066 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729"] Oct 30 05:02:43.439365 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.439339 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-fcd6a6-openshift-default-7b9d84db75-z6729"] Oct 30 05:02:43.927651 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.927619 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd"] Oct 30 05:02:43.927819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:43.927798 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" podUID="3ceb7e2e-f048-4520-93e1-7438a80fba50" containerName="istio-proxy" containerID="cri-o://ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19" gracePeriod=30 Oct 30 05:02:44.442935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:44.442904 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:44.848229 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:44.848130 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" path="/var/lib/kubelet/pods/b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc/volumes" Oct 30 05:02:46.228202 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.228166 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:46.909246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.909221 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 05:02:46.981971 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.981931 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-certs\") pod \"7e58a7bb-c456-4636-b455-b331d9185f70\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " Oct 30 05:02:46.982189 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.982010 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-socket\") pod \"7e58a7bb-c456-4636-b455-b331d9185f70\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " Oct 30 05:02:46.982189 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.982067 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-envoy\") pod \"7e58a7bb-c456-4636-b455-b331d9185f70\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " Oct 30 05:02:46.982189 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.982100 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e58a7bb-c456-4636-b455-b331d9185f70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e58a7bb-c456-4636-b455-b331d9185f70-istiod-ca-cert\") pod \"7e58a7bb-c456-4636-b455-b331d9185f70\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " Oct 30 05:02:46.982189 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.982167 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e58a7bb-c456-4636-b455-b331d9185f70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e58a7bb-c456-4636-b455-b331d9185f70-istio-podinfo\") pod \"7e58a7bb-c456-4636-b455-b331d9185f70\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " Oct 30 05:02:46.982516 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.982201 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-credential-socket\") pod \"7e58a7bb-c456-4636-b455-b331d9185f70\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " Oct 30 05:02:46.982516 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.982471 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-kube-api-access-7h9tn\" (UniqueName: \"kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-kube-api-access-7h9tn\") pod \"7e58a7bb-c456-4636-b455-b331d9185f70\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " Oct 30 05:02:46.982516 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.982514 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-data\") pod \"7e58a7bb-c456-4636-b455-b331d9185f70\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " Oct 30 05:02:46.982714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.982547 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-istio-token\" (UniqueName: \"kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-istio-token\") pod \"7e58a7bb-c456-4636-b455-b331d9185f70\" (UID: \"7e58a7bb-c456-4636-b455-b331d9185f70\") " Oct 30 05:02:46.984457 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.983972 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e58a7bb-c456-4636-b455-b331d9185f70-istiod-ca-cert" pod "7e58a7bb-c456-4636-b455-b331d9185f70" (UID: "7e58a7bb-c456-4636-b455-b331d9185f70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:46.984457 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.984206 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-certs" pod "7e58a7bb-c456-4636-b455-b331d9185f70" (UID: "7e58a7bb-c456-4636-b455-b331d9185f70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:46.984457 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.984363 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-socket" pod "7e58a7bb-c456-4636-b455-b331d9185f70" (UID: "7e58a7bb-c456-4636-b455-b331d9185f70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:46.985450 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.985406 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-credential-socket" pod "7e58a7bb-c456-4636-b455-b331d9185f70" (UID: "7e58a7bb-c456-4636-b455-b331d9185f70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:46.987011 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:46.986988 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-data" pod "7e58a7bb-c456-4636-b455-b331d9185f70" (UID: "7e58a7bb-c456-4636-b455-b331d9185f70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:47.006899 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.006858 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e58a7bb-c456-4636-b455-b331d9185f70-istio-podinfo" pod "7e58a7bb-c456-4636-b455-b331d9185f70" (UID: "7e58a7bb-c456-4636-b455-b331d9185f70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:02:47.007162 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.007131 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-envoy" pod "7e58a7bb-c456-4636-b455-b331d9185f70" (UID: "7e58a7bb-c456-4636-b455-b331d9185f70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:47.008646 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.008623 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-istio-token" pod "7e58a7bb-c456-4636-b455-b331d9185f70" (UID: "7e58a7bb-c456-4636-b455-b331d9185f70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:47.009522 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.009488 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-kube-api-access-7h9tn" pod "7e58a7bb-c456-4636-b455-b331d9185f70" (UID: "7e58a7bb-c456-4636-b455-b331d9185f70"). InnerVolumeSpecName "kube-api-access-7h9tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:47.084781 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.084749 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h9tn\" (UniqueName: \"kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-kube-api-access-7h9tn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:47.084781 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.084779 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:47.084973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.084789 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e58a7bb-c456-4636-b455-b331d9185f70-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:47.084973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.084799 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:47.084973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.084808 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:47.084973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.084818 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:47.084973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.084826 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e58a7bb-c456-4636-b455-b331d9185f70-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:47.084973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.084834 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e58a7bb-c456-4636-b455-b331d9185f70-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:47.084973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.084843 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e58a7bb-c456-4636-b455-b331d9185f70-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:47.128372 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.128333 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:47.372444 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.369569 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:47.473754 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.473717 4793 generic.go:356] "Generic (PLEG): container finished" podID="7e58a7bb-c456-4636-b455-b331d9185f70" containerID="df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f" exitCode=0 Oct 30 05:02:47.474098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.474077 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" event={"ID":"7e58a7bb-c456-4636-b455-b331d9185f70","Type":"ContainerDied","Data":"df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f"} Oct 30 05:02:47.474246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.474230 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" event={"ID":"7e58a7bb-c456-4636-b455-b331d9185f70","Type":"ContainerDied","Data":"14a9d6b999722aaa572cc32b4fba9813cd4c843afd9dd39efb4e8a749c7b2177"} Oct 30 05:02:47.474371 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.474358 4793 scope.go:122] "RemoveContainer" containerID="df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f" Oct 30 05:02:47.474645 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.474627 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts" Oct 30 05:02:47.516815 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.516789 4793 scope.go:122] "RemoveContainer" containerID="df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f" Oct 30 05:02:47.518584 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:02:47.518556 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f\": container with ID starting with df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f not found: ID does not exist" containerID="df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f" Oct 30 05:02:47.518806 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.518782 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f"} err="failed to get container status \"df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f\": rpc error: code = NotFound desc = could not find container \"df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f\": container with ID starting with df2a78ccea67bead7ddc1cfc724432feb69126896fb8a8bd768af02aa6ab039f not found: ID does not exist" Oct 30 05:02:47.619446 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.617813 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts"] Oct 30 05:02:47.639587 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.639504 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-v45ts"] Oct 30 05:02:47.672335 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.672286 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:47.712794 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.709963 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b"] Oct 30 05:02:47.712794 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.710196 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" podUID="3083e820-94e6-4000-8987-a57f3bd03798" containerName="maas-api" containerID="cri-o://7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af" gracePeriod=30 Oct 30 05:02:47.885455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:47.885390 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:48.068311 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.067045 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:02:48.068311 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.067123 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" podUID="3ceb7e2e-f048-4520-93e1-7438a80fba50" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:02:48.363963 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.363938 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 05:02:48.484737 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.484702 4793 generic.go:356] "Generic (PLEG): container finished" podID="3083e820-94e6-4000-8987-a57f3bd03798" containerID="7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af" exitCode=0 Oct 30 05:02:48.485236 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.484770 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" Oct 30 05:02:48.485236 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.484782 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" event={"ID":"3083e820-94e6-4000-8987-a57f3bd03798","Type":"ContainerDied","Data":"7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af"} Oct 30 05:02:48.485236 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.484859 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b" event={"ID":"3083e820-94e6-4000-8987-a57f3bd03798","Type":"ContainerDied","Data":"523a945010b76ca3d23bda3006b8f53955562f52a29dc59eba6a849b157c0e88"} Oct 30 05:02:48.485236 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.484894 4793 scope.go:122] "RemoveContainer" containerID="7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af" Oct 30 05:02:48.493730 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.493706 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-metrics-certs\") pod \"3083e820-94e6-4000-8987-a57f3bd03798\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " Oct 30 05:02:48.493877 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.493811 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3083e820-94e6-4000-8987-a57f3bd03798-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3083e820-94e6-4000-8987-a57f3bd03798-openshift-service-ca\") pod \"3083e820-94e6-4000-8987-a57f3bd03798\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " Oct 30 05:02:48.493877 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.493851 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-maas-api-tls\") pod \"3083e820-94e6-4000-8987-a57f3bd03798\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " Oct 30 05:02:48.494059 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.494042 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3083e820-94e6-4000-8987-a57f3bd03798-kube-api-access-bcnft\" (UniqueName: \"kubernetes.io/projected/3083e820-94e6-4000-8987-a57f3bd03798-kube-api-access-bcnft\") pod \"3083e820-94e6-4000-8987-a57f3bd03798\" (UID: \"3083e820-94e6-4000-8987-a57f3bd03798\") " Oct 30 05:02:48.494233 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.494207 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3083e820-94e6-4000-8987-a57f3bd03798-openshift-service-ca" pod "3083e820-94e6-4000-8987-a57f3bd03798" (UID: "3083e820-94e6-4000-8987-a57f3bd03798"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:48.495008 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.494981 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3083e820-94e6-4000-8987-a57f3bd03798-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:48.496002 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.495980 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-metrics-certs" pod "3083e820-94e6-4000-8987-a57f3bd03798" (UID: "3083e820-94e6-4000-8987-a57f3bd03798"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:48.496572 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.496544 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-maas-api-tls" pod "3083e820-94e6-4000-8987-a57f3bd03798" (UID: "3083e820-94e6-4000-8987-a57f3bd03798"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:02:48.496687 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.496612 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3083e820-94e6-4000-8987-a57f3bd03798-kube-api-access-bcnft" pod "3083e820-94e6-4000-8987-a57f3bd03798" (UID: "3083e820-94e6-4000-8987-a57f3bd03798"). InnerVolumeSpecName "kube-api-access-bcnft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:48.498178 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.498161 4793 scope.go:122] "RemoveContainer" containerID="7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af" Oct 30 05:02:48.498413 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:02:48.498393 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af\": container with ID starting with 7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af not found: ID does not exist" containerID="7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af" Oct 30 05:02:48.498508 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.498435 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af"} err="failed to get container status \"7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af\": rpc error: code = NotFound desc = could not find container \"7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af\": container with ID starting with 7374211b4eb50da19f5223524a86a1f302a5293c1f7b756019ae0d63dc2789af not found: ID does not exist" Oct 30 05:02:48.597277 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.597189 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:48.597277 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.597229 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3083e820-94e6-4000-8987-a57f3bd03798-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:48.597277 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.597245 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcnft\" (UniqueName: \"kubernetes.io/projected/3083e820-94e6-4000-8987-a57f3bd03798-kube-api-access-bcnft\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:48.612708 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.612667 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:48.835298 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.835248 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b"] Oct 30 05:02:48.849327 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.849265 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" path="/var/lib/kubelet/pods/7e58a7bb-c456-4636-b455-b331d9185f70/volumes" Oct 30 05:02:48.849696 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:48.849677 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f9cd17-6f8b89b59f-6gg6b"] Oct 30 05:02:49.122264 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.122242 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" Oct 30 05:02:49.220878 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.220846 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 05:02:49.306924 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.306886 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-socket\") pod \"3ceb7e2e-f048-4520-93e1-7438a80fba50\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " Oct 30 05:02:49.307161 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.306977 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-envoy\") pod \"3ceb7e2e-f048-4520-93e1-7438a80fba50\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " Oct 30 05:02:49.307259 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.307237 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-socket" pod "3ceb7e2e-f048-4520-93e1-7438a80fba50" (UID: "3ceb7e2e-f048-4520-93e1-7438a80fba50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:49.307787 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.307761 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-kube-api-access-d9hl2\" (UniqueName: \"kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-kube-api-access-d9hl2\") pod \"3ceb7e2e-f048-4520-93e1-7438a80fba50\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " Oct 30 05:02:49.307999 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.307969 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-kube-api-access-z4jkq\" (UniqueName: \"kubernetes.io/projected/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-kube-api-access-z4jkq\") pod \"c658db96-88c7-42b3-9cf3-5fcc5c45a21e\" (UID: \"c658db96-88c7-42b3-9cf3-5fcc5c45a21e\") " Oct 30 05:02:49.308124 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.308060 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-plugins-config-volume\") pod \"c658db96-88c7-42b3-9cf3-5fcc5c45a21e\" (UID: \"c658db96-88c7-42b3-9cf3-5fcc5c45a21e\") " Oct 30 05:02:49.308202 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.308176 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-podinfo\") pod \"3ceb7e2e-f048-4520-93e1-7438a80fba50\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " Oct 30 05:02:49.308270 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.308225 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ceb7e2e-f048-4520-93e1-7438a80fba50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb7e2e-f048-4520-93e1-7438a80fba50-istiod-ca-cert\") pod \"3ceb7e2e-f048-4520-93e1-7438a80fba50\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " Oct 30 05:02:49.308270 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.308255 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-data\") pod \"3ceb7e2e-f048-4520-93e1-7438a80fba50\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " Oct 30 05:02:49.308402 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.308306 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-credential-socket\") pod \"3ceb7e2e-f048-4520-93e1-7438a80fba50\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " Oct 30 05:02:49.308402 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.308350 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-certs\") pod \"3ceb7e2e-f048-4520-93e1-7438a80fba50\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " Oct 30 05:02:49.308402 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.308378 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-token\") pod \"3ceb7e2e-f048-4520-93e1-7438a80fba50\" (UID: \"3ceb7e2e-f048-4520-93e1-7438a80fba50\") " Oct 30 05:02:49.308971 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.308950 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:49.309850 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.309552 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-credential-socket" pod "3ceb7e2e-f048-4520-93e1-7438a80fba50" (UID: "3ceb7e2e-f048-4520-93e1-7438a80fba50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:49.309850 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.309626 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ceb7e2e-f048-4520-93e1-7438a80fba50-istiod-ca-cert" pod "3ceb7e2e-f048-4520-93e1-7438a80fba50" (UID: "3ceb7e2e-f048-4520-93e1-7438a80fba50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:49.309850 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.309778 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-data" pod "3ceb7e2e-f048-4520-93e1-7438a80fba50" (UID: "3ceb7e2e-f048-4520-93e1-7438a80fba50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:49.310474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.309916 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-envoy" pod "3ceb7e2e-f048-4520-93e1-7438a80fba50" (UID: "3ceb7e2e-f048-4520-93e1-7438a80fba50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:49.310474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.310125 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-certs" pod "3ceb7e2e-f048-4520-93e1-7438a80fba50" (UID: "3ceb7e2e-f048-4520-93e1-7438a80fba50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:02:49.310474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.310354 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-plugins-config-volume" pod "c658db96-88c7-42b3-9cf3-5fcc5c45a21e" (UID: "c658db96-88c7-42b3-9cf3-5fcc5c45a21e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:02:49.310671 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.310563 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-kube-api-access-d9hl2" pod "3ceb7e2e-f048-4520-93e1-7438a80fba50" (UID: "3ceb7e2e-f048-4520-93e1-7438a80fba50"). InnerVolumeSpecName "kube-api-access-d9hl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:49.311735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.311684 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-token" pod "3ceb7e2e-f048-4520-93e1-7438a80fba50" (UID: "3ceb7e2e-f048-4520-93e1-7438a80fba50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:49.312031 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.312007 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-kube-api-access-z4jkq" pod "c658db96-88c7-42b3-9cf3-5fcc5c45a21e" (UID: "c658db96-88c7-42b3-9cf3-5fcc5c45a21e"). InnerVolumeSpecName "kube-api-access-z4jkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:02:49.312248 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.312227 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-podinfo" pod "3ceb7e2e-f048-4520-93e1-7438a80fba50" (UID: "3ceb7e2e-f048-4520-93e1-7438a80fba50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:02:49.410393 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.410303 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4jkq\" (UniqueName: \"kubernetes.io/projected/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-kube-api-access-z4jkq\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:49.410393 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.410339 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c658db96-88c7-42b3-9cf3-5fcc5c45a21e-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:49.410393 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.410354 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:49.410393 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.410367 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ceb7e2e-f048-4520-93e1-7438a80fba50-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:49.410393 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.410380 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:49.410832 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.410393 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:49.410832 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.410430 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:49.410832 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.410444 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:49.410832 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.410457 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ceb7e2e-f048-4520-93e1-7438a80fba50-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:49.410832 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.410470 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9hl2\" (UniqueName: \"kubernetes.io/projected/3ceb7e2e-f048-4520-93e1-7438a80fba50-kube-api-access-d9hl2\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:02:49.506091 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.505982 4793 generic.go:356] "Generic (PLEG): container finished" podID="3ceb7e2e-f048-4520-93e1-7438a80fba50" containerID="ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19" exitCode=0 Oct 30 05:02:49.506628 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.506185 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" event={"ID":"3ceb7e2e-f048-4520-93e1-7438a80fba50","Type":"ContainerDied","Data":"ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19"} Oct 30 05:02:49.506628 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.506282 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" event={"ID":"3ceb7e2e-f048-4520-93e1-7438a80fba50","Type":"ContainerDied","Data":"81874562f8dddb315cd80bb79ca79eb92b56641f5f5b73c2f504f7f1ba0eb9df"} Oct 30 05:02:49.506628 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.506323 4793 scope.go:122] "RemoveContainer" containerID="ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19" Oct 30 05:02:49.506628 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.506587 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd" Oct 30 05:02:49.509967 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.509937 4793 generic.go:356] "Generic (PLEG): container finished" podID="c658db96-88c7-42b3-9cf3-5fcc5c45a21e" containerID="ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7" exitCode=0 Oct 30 05:02:49.510354 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.510327 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" event={"ID":"c658db96-88c7-42b3-9cf3-5fcc5c45a21e","Type":"ContainerDied","Data":"ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7"} Oct 30 05:02:49.510634 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.510581 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" event={"ID":"c658db96-88c7-42b3-9cf3-5fcc5c45a21e","Type":"ContainerDied","Data":"29e543232b5cfeb52bafeae12c0566ee03bb68ee1806c2142effc530f37bd36c"} Oct 30 05:02:49.510916 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.510897 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m" Oct 30 05:02:49.531690 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.531486 4793 scope.go:122] "RemoveContainer" containerID="ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19" Oct 30 05:02:49.531857 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:02:49.531784 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19\": container with ID starting with ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19 not found: ID does not exist" containerID="ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19" Oct 30 05:02:49.531857 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.531820 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19"} err="failed to get container status \"ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19\": rpc error: code = NotFound desc = could not find container \"ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19\": container with ID starting with ee77cb7ad39cf7700e8c65ece43b411c5e03412c0dabfb28db081467f95dea19 not found: ID does not exist" Oct 30 05:02:49.531857 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.531838 4793 scope.go:122] "RemoveContainer" containerID="ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7" Oct 30 05:02:49.544805 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.544787 4793 scope.go:122] "RemoveContainer" containerID="ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7" Oct 30 05:02:49.545149 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:02:49.545127 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7\": container with ID starting with ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7 not found: ID does not exist" containerID="ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7" Oct 30 05:02:49.545246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.545160 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7"} err="failed to get container status \"ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7\": rpc error: code = NotFound desc = could not find container \"ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7\": container with ID starting with ef28fad5f9cdaf17cf6b343a43451861833a54880526736e862e0fdd2d6786d7 not found: ID does not exist" Oct 30 05:02:49.577184 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.577155 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m"] Oct 30 05:02:49.583743 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.583717 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82bec5-6f859cdf68-8pb9m"] Oct 30 05:02:49.616582 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.616559 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd"] Oct 30 05:02:49.620859 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:49.620837 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-495ebe-openshift-default-5f966f7587-p2bwd"] Oct 30 05:02:50.304849 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:50.304817 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:50.848790 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:50.848750 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3083e820-94e6-4000-8987-a57f3bd03798" path="/var/lib/kubelet/pods/3083e820-94e6-4000-8987-a57f3bd03798/volumes" Oct 30 05:02:50.849464 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:50.849444 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ceb7e2e-f048-4520-93e1-7438a80fba50" path="/var/lib/kubelet/pods/3ceb7e2e-f048-4520-93e1-7438a80fba50/volumes" Oct 30 05:02:50.849916 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:50.849895 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c658db96-88c7-42b3-9cf3-5fcc5c45a21e" path="/var/lib/kubelet/pods/c658db96-88c7-42b3-9cf3-5fcc5c45a21e/volumes" Oct 30 05:02:51.936086 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:51.936044 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:53.740271 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:53.740235 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:53.968864 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:53.968819 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:55.402034 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:55.401995 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:57.108445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:57.108386 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:57.967440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:57.967389 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:58.353784 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:58.353685 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:02:58.865246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:02:58.865209 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:00.716396 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:00.716363 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:02.398397 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:02.398362 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:04.019401 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:04.019364 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:05.716905 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:05.716866 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:07.369981 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:07.369936 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:08.976972 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:08.976931 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:10.634429 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:10.634330 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:11.219604 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:03:11.219565 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:03:12.321021 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:12.320986 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:13.701848 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:13.701803 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:14.466909 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:14.466875 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:16.128297 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:16.128267 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:17.854411 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:17.854373 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:19.419276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:19.419234 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:21.082459 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:21.082404 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:22.070014 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:22.069972 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:22.336320 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:22.336234 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:22.847082 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:22.847045 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:23.090837 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:23.090801 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:23.143910 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:23.143815 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7"] Oct 30 05:03:23.144101 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:23.144075 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" podUID="0a966611-bdd4-4210-8908-a1e3eee59590" containerName="istio-proxy" containerID="cri-o://dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432" gracePeriod=30 Oct 30 05:03:23.395167 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:23.394976 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:24.240968 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:24.240934 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:24.548648 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:24.548559 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:25.022460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:25.022387 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:26.717516 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:26.717475 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:28.433010 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.432970 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:28.476737 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.476710 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:03:28.578992 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.578940 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-data\") pod \"0a966611-bdd4-4210-8908-a1e3eee59590\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " Oct 30 05:03:28.579206 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.579190 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-certs\") pod \"0a966611-bdd4-4210-8908-a1e3eee59590\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " Oct 30 05:03:28.579286 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.579239 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-kube-api-access-8gj9r\" (UniqueName: \"kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-kube-api-access-8gj9r\") pod \"0a966611-bdd4-4210-8908-a1e3eee59590\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " Oct 30 05:03:28.579286 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.579266 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a966611-bdd4-4210-8908-a1e3eee59590-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a966611-bdd4-4210-8908-a1e3eee59590-istiod-ca-cert\") pod \"0a966611-bdd4-4210-8908-a1e3eee59590\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " Oct 30 05:03:28.579406 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.579388 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-istio-token\" (UniqueName: \"kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-istio-token\") pod \"0a966611-bdd4-4210-8908-a1e3eee59590\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " Oct 30 05:03:28.579646 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.579451 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-envoy\") pod \"0a966611-bdd4-4210-8908-a1e3eee59590\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " Oct 30 05:03:28.579646 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.579507 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a966611-bdd4-4210-8908-a1e3eee59590-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a966611-bdd4-4210-8908-a1e3eee59590-istio-podinfo\") pod \"0a966611-bdd4-4210-8908-a1e3eee59590\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " Oct 30 05:03:28.579646 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.579561 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-credential-socket\") pod \"0a966611-bdd4-4210-8908-a1e3eee59590\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " Oct 30 05:03:28.579646 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.579612 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-socket\") pod \"0a966611-bdd4-4210-8908-a1e3eee59590\" (UID: \"0a966611-bdd4-4210-8908-a1e3eee59590\") " Oct 30 05:03:28.580232 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.580202 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-certs" pod "0a966611-bdd4-4210-8908-a1e3eee59590" (UID: "0a966611-bdd4-4210-8908-a1e3eee59590"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:03:28.580353 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.580337 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:28.580477 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.580443 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-socket" pod "0a966611-bdd4-4210-8908-a1e3eee59590" (UID: "0a966611-bdd4-4210-8908-a1e3eee59590"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:03:28.580589 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.580570 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-data" pod "0a966611-bdd4-4210-8908-a1e3eee59590" (UID: "0a966611-bdd4-4210-8908-a1e3eee59590"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:03:28.580761 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.580736 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a966611-bdd4-4210-8908-a1e3eee59590-istiod-ca-cert" pod "0a966611-bdd4-4210-8908-a1e3eee59590" (UID: "0a966611-bdd4-4210-8908-a1e3eee59590"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:03:28.581035 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.581009 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-credential-socket" pod "0a966611-bdd4-4210-8908-a1e3eee59590" (UID: "0a966611-bdd4-4210-8908-a1e3eee59590"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:03:28.583408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.583382 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a966611-bdd4-4210-8908-a1e3eee59590-istio-podinfo" pod "0a966611-bdd4-4210-8908-a1e3eee59590" (UID: "0a966611-bdd4-4210-8908-a1e3eee59590"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:03:28.583790 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.583765 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-istio-token" pod "0a966611-bdd4-4210-8908-a1e3eee59590" (UID: "0a966611-bdd4-4210-8908-a1e3eee59590"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:03:28.584083 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.584048 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-kube-api-access-8gj9r" pod "0a966611-bdd4-4210-8908-a1e3eee59590" (UID: "0a966611-bdd4-4210-8908-a1e3eee59590"). InnerVolumeSpecName "kube-api-access-8gj9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:03:28.585300 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.585271 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-envoy" pod "0a966611-bdd4-4210-8908-a1e3eee59590" (UID: "0a966611-bdd4-4210-8908-a1e3eee59590"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:03:28.681731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.681696 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gj9r\" (UniqueName: \"kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-kube-api-access-8gj9r\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:28.681731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.681723 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a966611-bdd4-4210-8908-a1e3eee59590-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:28.681731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.681734 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a966611-bdd4-4210-8908-a1e3eee59590-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:28.682009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.681744 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:28.682009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.681752 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a966611-bdd4-4210-8908-a1e3eee59590-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:28.682009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.681760 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:28.682009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.681771 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:28.682009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.681781 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a966611-bdd4-4210-8908-a1e3eee59590-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:28.878190 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.878152 4793 generic.go:356] "Generic (PLEG): container finished" podID="0a966611-bdd4-4210-8908-a1e3eee59590" containerID="dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432" exitCode=0 Oct 30 05:03:28.878374 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.878266 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" event={"ID":"0a966611-bdd4-4210-8908-a1e3eee59590","Type":"ContainerDied","Data":"dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432"} Oct 30 05:03:28.878374 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.878279 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" Oct 30 05:03:28.878374 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.878317 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7" event={"ID":"0a966611-bdd4-4210-8908-a1e3eee59590","Type":"ContainerDied","Data":"f6b2dc32e2202816485d299f036c071261373d90edf50d1e710690745b195dd2"} Oct 30 05:03:28.878374 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.878369 4793 scope.go:122] "RemoveContainer" containerID="dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432" Oct 30 05:03:28.889927 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.889908 4793 scope.go:122] "RemoveContainer" containerID="dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432" Oct 30 05:03:28.890286 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:03:28.890262 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432\": container with ID starting with dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432 not found: ID does not exist" containerID="dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432" Oct 30 05:03:28.890371 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.890293 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432"} err="failed to get container status \"dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432\": rpc error: code = NotFound desc = could not find container \"dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432\": container with ID starting with dcf0b626dca798f359f7ea5f7fe217ab054243c9630f510ae4ab306a8ca1e432 not found: ID does not exist" Oct 30 05:03:28.932473 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.932448 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7"] Oct 30 05:03:28.943618 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:28.943595 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-82bec5-openshift-default-f944cddf5-r74s7"] Oct 30 05:03:30.009951 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:30.009918 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:30.849245 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:30.849213 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a966611-bdd4-4210-8908-a1e3eee59590" path="/var/lib/kubelet/pods/0a966611-bdd4-4210-8908-a1e3eee59590/volumes" Oct 30 05:03:31.706297 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:31.706254 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:32.092825 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:32.092740 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9"] Oct 30 05:03:32.256135 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:32.256087 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:33.509129 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:33.509084 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:35.323607 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:35.323572 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:37.162459 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:37.162002 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:38.923006 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:38.922973 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:40.841874 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:40.841837 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:41.220522 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:03:41.220483 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:03:42.652949 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:42.652832 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:44.437104 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:44.437067 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:45.707391 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:45.707356 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:45.934078 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:45.934039 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:46.197475 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:46.197401 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:46.412326 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:46.412296 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:46.633389 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:46.633297 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:47.124373 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.124335 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" podUID="667e7bbb-b453-4597-ad84-f529a4f8d8b1" containerName="main" containerID="cri-o://8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c" gracePeriod=45 Oct 30 05:03:47.434504 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.434478 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 05:03:47.505365 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.505323 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-dshm\") pod \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " Oct 30 05:03:47.505602 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.505395 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tls-certs\") pod \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " Oct 30 05:03:47.505602 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.505513 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-home\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-home\") pod \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " Oct 30 05:03:47.505731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.505600 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667e7bbb-b453-4597-ad84-f529a4f8d8b1-kube-api-access-b5x5r\" (UniqueName: \"kubernetes.io/projected/667e7bbb-b453-4597-ad84-f529a4f8d8b1-kube-api-access-b5x5r\") pod \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " Oct 30 05:03:47.505731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.505640 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tmp-dir\") pod \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " Oct 30 05:03:47.505875 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.505852 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-model-cache\") pod \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\" (UID: \"667e7bbb-b453-4597-ad84-f529a4f8d8b1\") " Oct 30 05:03:47.506193 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.506169 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-home" pod "667e7bbb-b453-4597-ad84-f529a4f8d8b1" (UID: "667e7bbb-b453-4597-ad84-f529a4f8d8b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:03:47.506529 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.506506 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-model-cache" pod "667e7bbb-b453-4597-ad84-f529a4f8d8b1" (UID: "667e7bbb-b453-4597-ad84-f529a4f8d8b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:03:47.506699 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.506673 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tmp-dir" pod "667e7bbb-b453-4597-ad84-f529a4f8d8b1" (UID: "667e7bbb-b453-4597-ad84-f529a4f8d8b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:03:47.506931 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.506908 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:47.507025 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.506935 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:47.507025 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.506948 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:47.509161 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.509137 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tls-certs" pod "667e7bbb-b453-4597-ad84-f529a4f8d8b1" (UID: "667e7bbb-b453-4597-ad84-f529a4f8d8b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:03:47.509373 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.509348 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667e7bbb-b453-4597-ad84-f529a4f8d8b1-kube-api-access-b5x5r" pod "667e7bbb-b453-4597-ad84-f529a4f8d8b1" (UID: "667e7bbb-b453-4597-ad84-f529a4f8d8b1"). InnerVolumeSpecName "kube-api-access-b5x5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:03:47.514569 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.514539 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-dshm" pod "667e7bbb-b453-4597-ad84-f529a4f8d8b1" (UID: "667e7bbb-b453-4597-ad84-f529a4f8d8b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:03:47.580267 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.580215 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:47.608913 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.608886 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/667e7bbb-b453-4597-ad84-f529a4f8d8b1-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:47.608913 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.608913 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/667e7bbb-b453-4597-ad84-f529a4f8d8b1-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:47.609097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.608925 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5x5r\" (UniqueName: \"kubernetes.io/projected/667e7bbb-b453-4597-ad84-f529a4f8d8b1-kube-api-access-b5x5r\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:03:47.833019 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:47.832988 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:48.045039 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:48.045002 4793 generic.go:356] "Generic (PLEG): container finished" podID="667e7bbb-b453-4597-ad84-f529a4f8d8b1" containerID="8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c" exitCode=0 Oct 30 05:03:48.045246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:48.045044 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" Oct 30 05:03:48.045246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:48.045068 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" event={"ID":"667e7bbb-b453-4597-ad84-f529a4f8d8b1","Type":"ContainerDied","Data":"8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c"} Oct 30 05:03:48.045246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:48.045107 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9" event={"ID":"667e7bbb-b453-4597-ad84-f529a4f8d8b1","Type":"ContainerDied","Data":"0158d17b3ea926687d5aaede5933a15dc5e13eede5d532aa38d5d3acf52f63f6"} Oct 30 05:03:48.045246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:48.045130 4793 scope.go:122] "RemoveContainer" containerID="8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c" Oct 30 05:03:48.058697 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:48.058679 4793 scope.go:122] "RemoveContainer" containerID="8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c" Oct 30 05:03:48.058969 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:03:48.058949 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c\": container with ID starting with 8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c not found: ID does not exist" containerID="8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c" Oct 30 05:03:48.059033 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:48.058976 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c"} err="failed to get container status \"8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c\": rpc error: code = NotFound desc = could not find container \"8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c\": container with ID starting with 8df8da07042de169dd42b50425c59f9547cb014be75f0f0e802acecf6ffe505c not found: ID does not exist" Oct 30 05:03:48.121769 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:48.121702 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9"] Oct 30 05:03:48.133268 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:48.133231 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-449d83/test-model-449d83-kserve-6f468d4bc-mdzc9"] Oct 30 05:03:48.445119 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:48.445087 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:48.662136 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:48.662105 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:48.849488 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:48.849395 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="667e7bbb-b453-4597-ad84-f529a4f8d8b1" path="/var/lib/kubelet/pods/667e7bbb-b453-4597-ad84-f529a4f8d8b1/volumes" Oct 30 05:03:50.161229 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:50.161191 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:51.939094 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:51.939061 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:52.259645 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:52.259534 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:52.756763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:52.756716 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:53.738219 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:53.738183 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:55.368815 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:55.368770 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:57.144699 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:57.144660 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:58.083333 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:58.083295 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:58.560741 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:58.560606 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:58.776302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:58.776269 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:59.076171 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:59.076086 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:03:59.301274 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:03:59.301237 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:00.786261 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:00.786227 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:02.497490 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:02.497437 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:04.139162 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:04.139103 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:05.232921 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:05.232881 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:05.903569 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:05.903517 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:07.507825 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:07.507784 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:09.240627 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:09.240595 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:11.078018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:11.077970 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:11.220859 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:04:11.220824 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:04:12.812286 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:12.812246 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:14.527564 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:14.527530 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:14.746857 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:14.746807 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:16.318793 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:16.318762 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:18.000136 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:18.000102 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:19.719032 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:19.718994 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:20.965063 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:20.965018 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:21.389723 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:21.389615 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:22.999456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:22.999379 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:24.634206 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:24.634167 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:26.315582 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.315545 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:26.893735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.893705 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp"] Oct 30 05:04:26.894473 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894455 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f256eec9-b172-444a-b3fe-a311bf865e01" containerName="istio-proxy" Oct 30 05:04:26.894473 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894472 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f256eec9-b172-444a-b3fe-a311bf865e01" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894482 4793 cpu_manager.go:414] "containerMap: removing container" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894488 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894498 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3ceb7e2e-f048-4520-93e1-7438a80fba50" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894503 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ceb7e2e-f048-4520-93e1-7438a80fba50" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894514 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c658db96-88c7-42b3-9cf3-5fcc5c45a21e" containerName="payload-processing" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894518 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c658db96-88c7-42b3-9cf3-5fcc5c45a21e" containerName="payload-processing" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894533 4793 cpu_manager.go:414] "containerMap: removing container" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894540 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894565 4793 cpu_manager.go:414] "containerMap: removing container" podUID="6b6e51a1-7f4e-492f-92b1-b0025a43fef0" containerName="maas-api" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894570 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6e51a1-7f4e-492f-92b1-b0025a43fef0" containerName="maas-api" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894577 4793 cpu_manager.go:414] "containerMap: removing container" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894581 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894588 4793 cpu_manager.go:414] "containerMap: removing container" podUID="667e7bbb-b453-4597-ad84-f529a4f8d8b1" containerName="main" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894592 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667e7bbb-b453-4597-ad84-f529a4f8d8b1" containerName="main" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894599 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894604 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894611 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894618 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894631 4793 cpu_manager.go:414] "containerMap: removing container" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894637 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894644 4793 cpu_manager.go:414] "containerMap: removing container" podUID="0a966611-bdd4-4210-8908-a1e3eee59590" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894648 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a966611-bdd4-4210-8908-a1e3eee59590" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894659 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3083e820-94e6-4000-8987-a57f3bd03798" containerName="maas-api" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894664 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3083e820-94e6-4000-8987-a57f3bd03798" containerName="maas-api" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894681 4793 cpu_manager.go:414] "containerMap: removing container" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" Oct 30 05:04:26.894717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894686 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894813 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f256eec9-b172-444a-b3fe-a311bf865e01" containerName="istio-proxy" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894827 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894835 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ceb7e2e-f048-4520-93e1-7438a80fba50" containerName="istio-proxy" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894842 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="c658db96-88c7-42b3-9cf3-5fcc5c45a21e" containerName="payload-processing" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894851 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894864 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6e51a1-7f4e-492f-92b1-b0025a43fef0" containerName="maas-api" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894872 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51f0c70-e6a7-4bf3-a367-511a9e2b9ccc" containerName="istio-proxy" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894884 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="667e7bbb-b453-4597-ad84-f529a4f8d8b1" containerName="main" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894893 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894901 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3abc285-621f-4873-a77d-70d6f93a6c53" containerName="istio-proxy" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894909 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a966611-bdd4-4210-8908-a1e3eee59590" containerName="istio-proxy" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894920 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3083e820-94e6-4000-8987-a57f3bd03798" containerName="maas-api" Oct 30 05:04:26.896324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.894931 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e58a7bb-c456-4636-b455-b331d9185f70" containerName="istio-proxy" Oct 30 05:04:26.898860 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.898838 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:26.935052 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.935019 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp"] Oct 30 05:04:26.940018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.939987 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n"] Oct 30 05:04:26.941569 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.941543 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac37ac34-46aa-4ad9-889b-117548e46735" containerName="istio-proxy" Oct 30 05:04:26.946102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.946080 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:26.948585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.948549 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqcw\" (UniqueName: \"kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-kube-api-access-4rqcw\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:26.948703 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.948613 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:26.948703 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.948648 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:26.948703 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.948691 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:26.948888 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.948735 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:26.948888 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.948857 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d234ea-de77-44a4-bec1-0ed26e00a078-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:26.948999 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.948909 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:26.948999 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.948963 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:26.949111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.949010 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:26.949111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.949042 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:26.949111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.949083 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:26.949288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.949115 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:26.949288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.949169 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:26.949288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.949198 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:26.949288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.949237 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqdlx\" (UniqueName: \"kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-kube-api-access-fqdlx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:26.949288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.949277 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:26.949709 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.949307 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:26.949709 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.949382 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:26.979511 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:26.976954 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n"] Oct 30 05:04:27.050646 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.050621 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.050790 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.050655 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.050854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.050781 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.050854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.050829 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.050968 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.050877 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.050968 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.050914 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.051084 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.050974 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.051084 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.050984 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.051084 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051006 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.051084 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051039 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.051084 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051079 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqdlx\" (UniqueName: \"kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-kube-api-access-fqdlx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.051368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051116 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.051368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051136 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.051368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051212 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.051368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051256 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqcw\" (UniqueName: \"kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-kube-api-access-4rqcw\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.051368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051312 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.051368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051351 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.051368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051364 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.051783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051445 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.051783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051509 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.051783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051578 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.051783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051672 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.051783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051687 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d234ea-de77-44a4-bec1-0ed26e00a078-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.052054 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051311 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.052054 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051949 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.052054 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.051958 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.052232 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.052191 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.052461 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.052441 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d234ea-de77-44a4-bec1-0ed26e00a078-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.053533 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.053510 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.053701 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.053681 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.054112 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.054086 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.054369 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.054349 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.074210 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.074181 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.074302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.074272 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.074345 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.074311 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqdlx\" (UniqueName: \"kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-kube-api-access-fqdlx\") pod \"maas-default-gateway-openshift-default-7f694df8bc-wmnwp\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.075039 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.075018 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqcw\" (UniqueName: \"kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-kube-api-access-4rqcw\") pod \"maas-default-gateway-openshift-default-7f694df8bc-gw78n\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.215469 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.215444 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:27.286677 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.286652 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:27.541593 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:04:27.541564 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d234ea_de77_44a4_bec1_0ed26e00a078.slice/crio-6e44090af1a28d23bdcd949768662b983789ae3bb3617e9f49e3a179f613e2ce WatchSource:0}: Error finding container 6e44090af1a28d23bdcd949768662b983789ae3bb3617e9f49e3a179f613e2ce: Status 404 returned error can't find the container with id 6e44090af1a28d23bdcd949768662b983789ae3bb3617e9f49e3a179f613e2ce Oct 30 05:04:27.542325 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.542298 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp"] Oct 30 05:04:27.544517 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.544484 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:04:27.544604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.544553 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:04:27.544604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.544587 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:04:27.656616 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.655203 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n"] Oct 30 05:04:27.659645 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.659607 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:04:27.659763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.659677 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:04:27.659763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:27.659708 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:04:28.000357 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:28.000325 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:28.440285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:28.440244 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" event={"ID":"a7d234ea-de77-44a4-bec1-0ed26e00a078","Type":"ContainerStarted","Data":"e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4"} Oct 30 05:04:28.440498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:28.440300 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" event={"ID":"a7d234ea-de77-44a4-bec1-0ed26e00a078","Type":"ContainerStarted","Data":"6e44090af1a28d23bdcd949768662b983789ae3bb3617e9f49e3a179f613e2ce"} Oct 30 05:04:28.441967 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:28.441940 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" event={"ID":"dea2f874-fede-4a13-8cb3-0b17a46c19fd","Type":"ContainerStarted","Data":"c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e"} Oct 30 05:04:28.442104 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:28.441981 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" event={"ID":"dea2f874-fede-4a13-8cb3-0b17a46c19fd","Type":"ContainerStarted","Data":"5241d3ef116cf0443a7e0fc86a5e4f0689fcda229bfd06d136fc354513f1c5c0"} Oct 30 05:04:28.497189 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:28.497117 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" podStartSLOduration=2.497098283 podStartE2EDuration="2.497098283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:28.495919183 +0000 UTC m=+3991.781793958" watchObservedRunningTime="2026-10-30 05:04:28.497098283 +0000 UTC m=+3991.782973058" Oct 30 05:04:28.549443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:28.546316 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" podStartSLOduration=2.546294333 podStartE2EDuration="2.546294333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:04:28.53986051 +0000 UTC m=+3991.825735286" watchObservedRunningTime="2026-10-30 05:04:28.546294333 +0000 UTC m=+3991.832169109" Oct 30 05:04:29.218445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:29.217805 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:29.219763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:29.219733 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wmnwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 30 05:04:29.219910 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:29.219788 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" podUID="a7d234ea-de77-44a4-bec1-0ed26e00a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 30 05:04:29.290893 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:29.289834 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:29.302442 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:29.294536 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gw78n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:04:29.302442 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:29.294598 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 05:04:29.751586 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:29.751546 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:30.216442 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:30.215926 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wmnwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 30 05:04:30.216442 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:30.215988 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" podUID="a7d234ea-de77-44a4-bec1-0ed26e00a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 30 05:04:30.295450 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:30.292355 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gw78n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:04:30.295450 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:30.292442 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 05:04:31.218832 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:31.218793 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wmnwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 30 05:04:31.219378 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:31.219350 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" podUID="a7d234ea-de77-44a4-bec1-0ed26e00a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 30 05:04:31.296853 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:31.296817 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gw78n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:04:31.297136 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:31.297106 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 05:04:31.413981 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:31.413946 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:32.216852 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:32.216812 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wmnwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 30 05:04:32.217143 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:32.217112 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" podUID="a7d234ea-de77-44a4-bec1-0ed26e00a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 30 05:04:32.289173 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:32.289124 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gw78n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:04:32.289837 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:32.289790 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 05:04:33.125238 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:33.125199 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:33.216255 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:33.216221 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wmnwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 30 05:04:33.216483 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:33.216288 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" podUID="a7d234ea-de77-44a4-bec1-0ed26e00a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 30 05:04:33.290836 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:33.290801 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gw78n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:04:33.291400 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:33.291370 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 05:04:34.216629 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:34.216588 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wmnwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 30 05:04:34.216822 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:34.216658 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" podUID="a7d234ea-de77-44a4-bec1-0ed26e00a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 30 05:04:34.287351 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:34.287313 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gw78n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:04:34.287557 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:34.287367 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 05:04:34.827738 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:34.827700 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:35.216369 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:35.216330 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-wmnwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body= Oct 30 05:04:35.216597 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:35.216395 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" podUID="a7d234ea-de77-44a4-bec1-0ed26e00a078" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" Oct 30 05:04:35.287425 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:35.287388 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gw78n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 05:04:35.287608 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:35.287473 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 05:04:36.045056 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.045006 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf"] Oct 30 05:04:36.046313 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.046267 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" podUID="15998a1b-c496-47e0-8887-0315b0354f2b" containerName="maas-api" containerID="cri-o://88092f6fb138f236c60770bcb94adebb115a73fc097814337b8006d2a3a55605" gracePeriod=30 Oct 30 05:04:36.207357 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.207332 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:36.689687 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.675728 4793 generic.go:356] "Generic (PLEG): container finished" podID="15998a1b-c496-47e0-8887-0315b0354f2b" containerID="88092f6fb138f236c60770bcb94adebb115a73fc097814337b8006d2a3a55605" exitCode=0 Oct 30 05:04:36.689687 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.676041 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" event={"ID":"15998a1b-c496-47e0-8887-0315b0354f2b","Type":"ContainerDied","Data":"88092f6fb138f236c60770bcb94adebb115a73fc097814337b8006d2a3a55605"} Oct 30 05:04:36.740608 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.740581 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 05:04:36.865146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.864661 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15998a1b-c496-47e0-8887-0315b0354f2b-kube-api-access-xsmd5\" (UniqueName: \"kubernetes.io/projected/15998a1b-c496-47e0-8887-0315b0354f2b-kube-api-access-xsmd5\") pod \"15998a1b-c496-47e0-8887-0315b0354f2b\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " Oct 30 05:04:36.865146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.864769 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-maas-api-tls\") pod \"15998a1b-c496-47e0-8887-0315b0354f2b\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " Oct 30 05:04:36.865146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.864881 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15998a1b-c496-47e0-8887-0315b0354f2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15998a1b-c496-47e0-8887-0315b0354f2b-openshift-service-ca\") pod \"15998a1b-c496-47e0-8887-0315b0354f2b\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " Oct 30 05:04:36.865146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.865117 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-metrics-certs\") pod \"15998a1b-c496-47e0-8887-0315b0354f2b\" (UID: \"15998a1b-c496-47e0-8887-0315b0354f2b\") " Oct 30 05:04:36.871751 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.868250 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15998a1b-c496-47e0-8887-0315b0354f2b-openshift-service-ca" pod "15998a1b-c496-47e0-8887-0315b0354f2b" (UID: "15998a1b-c496-47e0-8887-0315b0354f2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:04:36.874893 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.874846 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-metrics-certs" pod "15998a1b-c496-47e0-8887-0315b0354f2b" (UID: "15998a1b-c496-47e0-8887-0315b0354f2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:04:36.875288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.875254 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-maas-api-tls" pod "15998a1b-c496-47e0-8887-0315b0354f2b" (UID: "15998a1b-c496-47e0-8887-0315b0354f2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:04:36.875588 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.875558 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15998a1b-c496-47e0-8887-0315b0354f2b-kube-api-access-xsmd5" pod "15998a1b-c496-47e0-8887-0315b0354f2b" (UID: "15998a1b-c496-47e0-8887-0315b0354f2b"). InnerVolumeSpecName "kube-api-access-xsmd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:36.924053 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.923975 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:36.967828 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.967695 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:36.967828 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.967729 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsmd5\" (UniqueName: \"kubernetes.io/projected/15998a1b-c496-47e0-8887-0315b0354f2b-kube-api-access-xsmd5\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:36.967828 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.967745 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/15998a1b-c496-47e0-8887-0315b0354f2b-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:36.967828 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:36.967761 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/15998a1b-c496-47e0-8887-0315b0354f2b-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:37.148693 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.148651 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:37.152106 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.152080 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:37.153558 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.153536 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:04:37.297465 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.297328 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-gw78n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:04:37.297465 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.297406 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:04:37.332495 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.332460 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:37.333232 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.333208 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:37.334092 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.334069 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:04:37.481837 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.481796 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:37.685084 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.685058 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" Oct 30 05:04:37.685280 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.685077 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf" event={"ID":"15998a1b-c496-47e0-8887-0315b0354f2b","Type":"ContainerDied","Data":"511c949dceed8807a4e033e8e379525660a2683fb59b68cb441f0450da299f7e"} Oct 30 05:04:37.685280 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.685172 4793 scope.go:122] "RemoveContainer" containerID="88092f6fb138f236c60770bcb94adebb115a73fc097814337b8006d2a3a55605" Oct 30 05:04:37.701167 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.701141 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:37.740208 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.740185 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf"] Oct 30 05:04:37.744469 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:37.744448 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-449d83-68765bd56b-d4zjf"] Oct 30 05:04:38.511686 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:38.511645 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:38.847876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:38.847804 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15998a1b-c496-47e0-8887-0315b0354f2b" path="/var/lib/kubelet/pods/15998a1b-c496-47e0-8887-0315b0354f2b/volumes" Oct 30 05:04:39.511665 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:39.511578 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:39.737884 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:39.737848 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:40.235245 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:40.235208 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:40.477610 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:40.477571 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:40.496837 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:40.496757 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb"] Oct 30 05:04:40.497020 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:40.496986 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" containerID="cri-o://9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0" gracePeriod=30 Oct 30 05:04:40.721327 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:40.719728 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:41.220866 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:04:41.220824 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:04:41.495344 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:41.495266 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:41.953396 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:41.953360 4793 patch_prober.go:29] interesting pod/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:04:41.953612 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:41.953443 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:04:42.097702 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:42.097668 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:43.826598 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:43.826551 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:44.371526 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:44.371489 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:44.585010 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:44.584975 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:45.520358 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.520328 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:45.803206 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.803180 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 05:04:45.813772 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.813744 4793 generic.go:356] "Generic (PLEG): container finished" podID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerID="9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0" exitCode=0 Oct 30 05:04:45.813955 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.813810 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" Oct 30 05:04:45.813955 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.813829 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" event={"ID":"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28","Type":"ContainerDied","Data":"9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0"} Oct 30 05:04:45.813955 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.813884 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb" event={"ID":"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28","Type":"ContainerDied","Data":"6467e8f29ed6f3f149cf44fd6047d952dafc51b1e720dbf707f2c983649a4d41"} Oct 30 05:04:45.813955 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.813917 4793 scope.go:122] "RemoveContainer" containerID="9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0" Oct 30 05:04:45.826023 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.826005 4793 scope.go:122] "RemoveContainer" containerID="7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698" Oct 30 05:04:45.835944 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.835925 4793 scope.go:122] "RemoveContainer" containerID="9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0" Oct 30 05:04:45.836227 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:04:45.836199 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0\": container with ID starting with 9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0 not found: ID does not exist" containerID="9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0" Oct 30 05:04:45.836315 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.836230 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0"} err="failed to get container status \"9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0\": rpc error: code = NotFound desc = could not find container \"9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0\": container with ID starting with 9cb1ba4acd7f254acea34d65d8995ce594d5a47ca5c465cec7a8f6d58cbbf8d0 not found: ID does not exist" Oct 30 05:04:45.836315 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.836252 4793 scope.go:122] "RemoveContainer" containerID="7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698" Oct 30 05:04:45.836538 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:04:45.836518 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698\": container with ID starting with 7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698 not found: ID does not exist" containerID="7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698" Oct 30 05:04:45.836618 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.836542 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698"} err="failed to get container status \"7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698\": rpc error: code = NotFound desc = could not find container \"7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698\": container with ID starting with 7612eb5df642ef2e073d3578267ad805f7ac5a2cf52ebca40e61a32e92024698 not found: ID does not exist" Oct 30 05:04:45.915874 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.915844 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-data\") pod \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " Oct 30 05:04:45.916046 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.915955 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-credential-socket\") pod \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " Oct 30 05:04:45.916046 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.915999 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-socket\") pod \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " Oct 30 05:04:45.916046 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.916046 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-envoy\") pod \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " Oct 30 05:04:45.916254 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.916186 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-data" pod "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" (UID: "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:45.916314 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.916297 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-credential-socket" pod "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" (UID: "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:45.916413 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.916387 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-socket" pod "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" (UID: "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:45.916548 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.916524 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-token\" (UniqueName: \"kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-token\") pod \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " Oct 30 05:04:45.916619 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.916571 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-certs\") pod \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " Oct 30 05:04:45.916619 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.916593 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istiod-ca-cert\") pod \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " Oct 30 05:04:45.916734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.916631 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-podinfo\") pod \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " Oct 30 05:04:45.916734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.916666 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-kube-api-access-bs7tm\" (UniqueName: \"kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-kube-api-access-bs7tm\") pod \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\" (UID: \"18b6d0eb-2e34-43b0-9b83-c448f4b0fc28\") " Oct 30 05:04:45.917797 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.917145 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istiod-ca-cert" pod "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" (UID: "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:04:45.917797 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.917449 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:45.917797 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.917474 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:45.917797 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.917489 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:45.917797 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.917504 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:45.917797 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.917766 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-certs" pod "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" (UID: "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:45.919811 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.919785 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-envoy" pod "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" (UID: "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:04:45.919922 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.919843 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-podinfo" pod "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" (UID: "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:04:45.919999 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.919980 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-kube-api-access-bs7tm" pod "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" (UID: "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28"). InnerVolumeSpecName "kube-api-access-bs7tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:45.920036 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:45.920008 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-token" pod "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" (UID: "18b6d0eb-2e34-43b0-9b83-c448f4b0fc28"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:04:46.019262 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:46.019174 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:46.019262 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:46.019206 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:46.019262 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:46.019220 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:46.019262 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:46.019232 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs7tm\" (UniqueName: \"kubernetes.io/projected/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-kube-api-access-bs7tm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:46.019262 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:46.019244 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:04:46.163823 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:46.163787 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb"] Oct 30 05:04:46.175081 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:46.175054 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-449d83-openshift-default-67ccff74c5-rnqwb"] Oct 30 05:04:46.823122 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:46.822642 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:46.882798 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:46.882761 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" path="/var/lib/kubelet/pods/18b6d0eb-2e34-43b0-9b83-c448f4b0fc28/volumes" Oct 30 05:04:47.041702 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:47.041663 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:47.333812 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:47.333776 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:47.702959 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:47.702904 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:48.649342 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:48.649306 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:48.935196 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:48.935148 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:50.611750 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:50.611695 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:52.230294 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:52.230259 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:53.670394 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:53.670345 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:53.889355 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:53.889315 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:55.421863 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:55.421818 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:55.635913 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:55.635872 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:57.319902 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:57.319861 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:58.217563 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:58.217523 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:58.454213 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:58.454175 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:04:59.012516 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:04:59.012485 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:00.710140 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:00.710101 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:02.410674 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:02.410637 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:04.002831 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:04.002787 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:05.678341 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:05.678302 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:07.315600 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:07.315558 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:08.925436 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:08.925379 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:10.632883 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:10.632849 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:11.220787 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:05:11.220753 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:05:12.325174 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:12.325130 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:14.013559 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:14.013522 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:15.709886 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:15.709850 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:17.372456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:17.372400 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:18.957934 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:18.957895 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:20.602890 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:20.602845 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:22.199145 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:22.199109 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:23.935064 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:23.935027 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:25.651208 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:25.651166 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:27.394441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:27.387727 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:29.000865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:29.000810 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:30.720749 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:30.720709 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:31.881548 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:31.881509 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7"] Oct 30 05:05:32.031448 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.028283 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw"] Oct 30 05:05:32.039131 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.039092 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:32.054535 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.054491 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" podUID="893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf" containerName="maas-api" containerID="cri-o://e2d00a3b31c13b6a6a10e19e9595b96eb7297ebda51c2796c3895b722c27e2ef" gracePeriod=30 Oct 30 05:05:32.274961 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.274922 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:32.281883 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.281854 4793 generic.go:356] "Generic (PLEG): container finished" podID="893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf" containerID="e2d00a3b31c13b6a6a10e19e9595b96eb7297ebda51c2796c3895b722c27e2ef" exitCode=0 Oct 30 05:05:32.282039 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.281926 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" event={"ID":"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf","Type":"ContainerDied","Data":"e2d00a3b31c13b6a6a10e19e9595b96eb7297ebda51c2796c3895b722c27e2ef"} Oct 30 05:05:32.343150 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.343123 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 05:05:32.471653 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.471628 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-maas-api-tls\") pod \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " Oct 30 05:05:32.471783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.471705 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-kube-api-access-jzbhv\" (UniqueName: \"kubernetes.io/projected/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-kube-api-access-jzbhv\") pod \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " Oct 30 05:05:32.471829 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.471795 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-openshift-service-ca\") pod \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " Oct 30 05:05:32.471877 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.471827 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-metrics-certs\") pod \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\" (UID: \"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf\") " Oct 30 05:05:32.472164 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.472137 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-openshift-service-ca" pod "893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf" (UID: "893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:32.472683 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.472657 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:32.473910 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.473883 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-maas-api-tls" pod "893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf" (UID: "893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:05:32.474004 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.473943 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-metrics-certs" pod "893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf" (UID: "893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:05:32.474004 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.473942 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-kube-api-access-jzbhv" pod "893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf" (UID: "893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf"). InnerVolumeSpecName "kube-api-access-jzbhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:32.574075 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.574042 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:32.575488 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.575463 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzbhv\" (UniqueName: \"kubernetes.io/projected/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-kube-api-access-jzbhv\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:32.575684 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.575669 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:32.956923 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:32.956881 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:33.291832 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:33.291805 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" Oct 30 05:05:33.292030 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:33.291830 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw" event={"ID":"893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf","Type":"ContainerDied","Data":"32b28b087aace2a83cc205925bee0edcba5fcb5bbf08356e5bed324dfeda3358"} Oct 30 05:05:33.292030 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:33.291923 4793 scope.go:122] "RemoveContainer" containerID="e2d00a3b31c13b6a6a10e19e9595b96eb7297ebda51c2796c3895b722c27e2ef" Oct 30 05:05:33.346154 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:33.346124 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw"] Oct 30 05:05:33.357705 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:33.357679 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fd27a9-6574f8b6c4-6jmvw"] Oct 30 05:05:34.604963 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:34.604921 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:34.848930 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:34.848899 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf" path="/var/lib/kubelet/pods/893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf/volumes" Oct 30 05:05:35.476731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:35.476700 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:35.720207 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:35.720173 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:36.106161 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:36.106124 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:36.428491 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:36.426326 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:36.441516 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:36.441481 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j"] Oct 30 05:05:36.441962 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:36.441922 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" containerID="cri-o://5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928" gracePeriod=30 Oct 30 05:05:36.644915 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:36.644875 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:37.541981 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:37.541948 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:37.783944 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:37.783906 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:38.230247 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:38.230207 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:39.823054 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:39.823019 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:41.220604 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:05:41.220564 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:05:41.516752 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.516718 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:41.731336 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.731313 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 05:05:41.827159 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.827076 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-istio-token\" (UniqueName: \"kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-istio-token\") pod \"5f0fac68-9436-4c71-a3be-342e6054903e\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " Oct 30 05:05:41.827159 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.827141 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-envoy\") pod \"5f0fac68-9436-4c71-a3be-342e6054903e\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " Oct 30 05:05:41.827343 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.827173 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-kube-api-access-6nt9w\" (UniqueName: \"kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-kube-api-access-6nt9w\") pod \"5f0fac68-9436-4c71-a3be-342e6054903e\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " Oct 30 05:05:41.827343 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.827260 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-credential-socket\") pod \"5f0fac68-9436-4c71-a3be-342e6054903e\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " Oct 30 05:05:41.827461 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.827380 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-socket\") pod \"5f0fac68-9436-4c71-a3be-342e6054903e\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " Oct 30 05:05:41.827569 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.827535 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f0fac68-9436-4c71-a3be-342e6054903e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0fac68-9436-4c71-a3be-342e6054903e-istiod-ca-cert\") pod \"5f0fac68-9436-4c71-a3be-342e6054903e\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " Oct 30 05:05:41.827713 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.827589 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f0fac68-9436-4c71-a3be-342e6054903e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0fac68-9436-4c71-a3be-342e6054903e-istio-podinfo\") pod \"5f0fac68-9436-4c71-a3be-342e6054903e\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " Oct 30 05:05:41.827713 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.827674 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-data\") pod \"5f0fac68-9436-4c71-a3be-342e6054903e\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " Oct 30 05:05:41.827857 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.827712 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-certs\") pod \"5f0fac68-9436-4c71-a3be-342e6054903e\" (UID: \"5f0fac68-9436-4c71-a3be-342e6054903e\") " Oct 30 05:05:41.827943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.827899 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-credential-socket" pod "5f0fac68-9436-4c71-a3be-342e6054903e" (UID: "5f0fac68-9436-4c71-a3be-342e6054903e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:41.828132 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.828087 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-data" pod "5f0fac68-9436-4c71-a3be-342e6054903e" (UID: "5f0fac68-9436-4c71-a3be-342e6054903e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:41.828132 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.828112 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f0fac68-9436-4c71-a3be-342e6054903e-istiod-ca-cert" pod "5f0fac68-9436-4c71-a3be-342e6054903e" (UID: "5f0fac68-9436-4c71-a3be-342e6054903e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:05:41.828267 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.828157 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-certs" pod "5f0fac68-9436-4c71-a3be-342e6054903e" (UID: "5f0fac68-9436-4c71-a3be-342e6054903e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:41.828402 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.828279 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-socket" pod "5f0fac68-9436-4c71-a3be-342e6054903e" (UID: "5f0fac68-9436-4c71-a3be-342e6054903e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:41.828585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.828565 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f0fac68-9436-4c71-a3be-342e6054903e-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:41.828666 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.828590 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:41.828666 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.828603 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:41.828666 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.828616 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:41.828666 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.828632 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:41.829853 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.829826 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-istio-token" pod "5f0fac68-9436-4c71-a3be-342e6054903e" (UID: "5f0fac68-9436-4c71-a3be-342e6054903e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:41.830277 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.830253 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f0fac68-9436-4c71-a3be-342e6054903e-istio-podinfo" pod "5f0fac68-9436-4c71-a3be-342e6054903e" (UID: "5f0fac68-9436-4c71-a3be-342e6054903e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:05:41.830756 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.830734 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-kube-api-access-6nt9w" pod "5f0fac68-9436-4c71-a3be-342e6054903e" (UID: "5f0fac68-9436-4c71-a3be-342e6054903e"). InnerVolumeSpecName "kube-api-access-6nt9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:41.831819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.831797 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-envoy" pod "5f0fac68-9436-4c71-a3be-342e6054903e" (UID: "5f0fac68-9436-4c71-a3be-342e6054903e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:41.930553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.930526 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f0fac68-9436-4c71-a3be-342e6054903e-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:41.930553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.930551 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:41.930725 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.930578 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f0fac68-9436-4c71-a3be-342e6054903e-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:41.930725 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:41.930601 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nt9w\" (UniqueName: \"kubernetes.io/projected/5f0fac68-9436-4c71-a3be-342e6054903e-kube-api-access-6nt9w\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:42.116612 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.116533 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x"] Oct 30 05:05:42.117337 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117315 4793 cpu_manager.go:414] "containerMap: removing container" podUID="15998a1b-c496-47e0-8887-0315b0354f2b" containerName="maas-api" Oct 30 05:05:42.117337 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117336 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15998a1b-c496-47e0-8887-0315b0354f2b" containerName="maas-api" Oct 30 05:05:42.117493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117349 4793 cpu_manager.go:414] "containerMap: removing container" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" Oct 30 05:05:42.117493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117355 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" Oct 30 05:05:42.117493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117373 4793 cpu_manager.go:414] "containerMap: removing container" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" Oct 30 05:05:42.117493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117378 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" Oct 30 05:05:42.117493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117384 4793 cpu_manager.go:414] "containerMap: removing container" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" Oct 30 05:05:42.117493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117389 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" Oct 30 05:05:42.117493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117399 4793 cpu_manager.go:414] "containerMap: removing container" podUID="893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf" containerName="maas-api" Oct 30 05:05:42.117493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117404 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf" containerName="maas-api" Oct 30 05:05:42.117764 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117547 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" Oct 30 05:05:42.117764 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117558 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" Oct 30 05:05:42.117764 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117566 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" containerName="istio-proxy" Oct 30 05:05:42.117764 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117573 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="893d53f5-5c72-476e-a1d8-f2ed6ba5b9cf" containerName="maas-api" Oct 30 05:05:42.117764 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117580 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" Oct 30 05:05:42.117764 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.117593 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="15998a1b-c496-47e0-8887-0315b0354f2b" containerName="maas-api" Oct 30 05:05:42.121376 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.121358 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.150832 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.150803 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x"] Oct 30 05:05:42.234707 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.234680 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.235072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.234723 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.235072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.234747 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.235072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.234871 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.235072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.234937 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.235072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.235038 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.235260 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.235078 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2m95\" (UniqueName: \"kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-kube-api-access-q2m95\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.235260 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.235156 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.235260 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.235216 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.336955 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.336922 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.337152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.336991 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.337152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.337098 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.337152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.337144 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2m95\" (UniqueName: \"kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-kube-api-access-q2m95\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.337336 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.337197 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.337391 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.337326 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.337472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.337338 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.337517 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.337465 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.337562 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.337539 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.337604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.337577 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.337649 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.337607 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.337800 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.337771 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.337974 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.337953 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.338040 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.337958 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.339688 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.339669 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.339808 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.339791 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.359926 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.359892 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.360233 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.360216 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2m95\" (UniqueName: \"kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-kube-api-access-q2m95\") pod \"maas-default-gateway-openshift-default-7f694df8bc-vq58x\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.364109 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.364083 4793 generic.go:356] "Generic (PLEG): container finished" podID="5f0fac68-9436-4c71-a3be-342e6054903e" containerID="5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928" exitCode=0 Oct 30 05:05:42.364209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.364131 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" event={"ID":"5f0fac68-9436-4c71-a3be-342e6054903e","Type":"ContainerDied","Data":"5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928"} Oct 30 05:05:42.364209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.364138 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" Oct 30 05:05:42.364209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.364180 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j" event={"ID":"5f0fac68-9436-4c71-a3be-342e6054903e","Type":"ContainerDied","Data":"39ab53aa9fe9324734407dd500b538a9b7b8c5aa42b3a0834c5ab573bf2176b1"} Oct 30 05:05:42.364209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.364203 4793 scope.go:122] "RemoveContainer" containerID="5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928" Oct 30 05:05:42.377523 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.377506 4793 scope.go:122] "RemoveContainer" containerID="f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e" Oct 30 05:05:42.389659 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.389637 4793 scope.go:122] "RemoveContainer" containerID="5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928" Oct 30 05:05:42.389986 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:05:42.389965 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928\": container with ID starting with 5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928 not found: ID does not exist" containerID="5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928" Oct 30 05:05:42.390068 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.389994 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928"} err="failed to get container status \"5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928\": rpc error: code = NotFound desc = could not find container \"5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928\": container with ID starting with 5fda70b045e5972ed63cf44b09f7822e3f33b80488c374d480196db3e7fec928 not found: ID does not exist" Oct 30 05:05:42.390068 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.390012 4793 scope.go:122] "RemoveContainer" containerID="f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e" Oct 30 05:05:42.399883 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:05:42.392781 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e\": container with ID starting with f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e not found: ID does not exist" containerID="f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e" Oct 30 05:05:42.399883 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.392822 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e"} err="failed to get container status \"f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e\": rpc error: code = NotFound desc = could not find container \"f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e\": container with ID starting with f09dace91b16761a17045958966aee1086393a6fb506ece3bc32472575c2d66e not found: ID does not exist" Oct 30 05:05:42.418197 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.418170 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j"] Oct 30 05:05:42.422362 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.422332 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fd27a9-openshift-default-587d56b58c-2gv7j"] Oct 30 05:05:42.440802 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.440778 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:42.779238 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:05:42.779189 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4497e6f_eb1b_4b73_9bf8_c0de4bf5102e.slice/crio-ba65b4d4c8ee8b60e05864f1457e502ed55cb665788843f775c4080e78ee16b6 WatchSource:0}: Error finding container ba65b4d4c8ee8b60e05864f1457e502ed55cb665788843f775c4080e78ee16b6: Status 404 returned error can't find the container with id ba65b4d4c8ee8b60e05864f1457e502ed55cb665788843f775c4080e78ee16b6 Oct 30 05:05:42.780249 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.780215 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x"] Oct 30 05:05:42.783074 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.783031 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:05:42.783173 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.783119 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:05:42.783173 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.783162 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:05:42.848936 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:42.848911 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f0fac68-9436-4c71-a3be-342e6054903e" path="/var/lib/kubelet/pods/5f0fac68-9436-4c71-a3be-342e6054903e/volumes" Oct 30 05:05:43.235885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:43.235847 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:43.375198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:43.375161 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" event={"ID":"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e","Type":"ContainerStarted","Data":"03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c"} Oct 30 05:05:43.375382 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:43.375226 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" event={"ID":"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e","Type":"ContainerStarted","Data":"ba65b4d4c8ee8b60e05864f1457e502ed55cb665788843f775c4080e78ee16b6"} Oct 30 05:05:43.421671 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:43.421609 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" podStartSLOduration=1.42159293 podStartE2EDuration="1.42159293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:05:43.419012358 +0000 UTC m=+4066.704887133" watchObservedRunningTime="2026-10-30 05:05:43.42159293 +0000 UTC m=+4066.707467704" Oct 30 05:05:43.443478 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:43.441561 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:43.443478 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:43.442967 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vq58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:05:43.443478 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:43.443019 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 05:05:44.441758 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:44.441726 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vq58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:05:44.442153 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:44.441780 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 05:05:45.100441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:45.093754 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:45.314921 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:45.314874 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:45.441721 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:45.441690 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vq58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:05:45.441919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:45.441746 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 05:05:45.738812 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:45.738732 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:46.287953 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:46.287920 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:46.442072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:46.442038 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vq58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:05:46.442468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:46.442107 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 05:05:46.499899 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:46.499861 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:46.903602 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:46.903534 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" podUID="59ee8b24-fe74-4b1f-9f63-027e5fa5ec57" containerName="main" containerID="cri-o://8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35" gracePeriod=45 Oct 30 05:05:47.216092 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.215752 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:47.300227 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.300172 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:05:47.416723 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.416686 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-kube-api-access-mvq8s\" (UniqueName: \"kubernetes.io/projected/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-kube-api-access-mvq8s\") pod \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " Oct 30 05:05:47.416928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.416772 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tmp-dir\") pod \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " Oct 30 05:05:47.417006 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.416954 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-model-cache\") pod \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " Oct 30 05:05:47.417070 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.417035 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tls-certs\" (UniqueName: \"kubernetes.io/secret/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tls-certs\") pod \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " Oct 30 05:05:47.417132 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.417112 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-home\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-home\") pod \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " Oct 30 05:05:47.417183 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.417140 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-dshm\") pod \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\" (UID: \"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57\") " Oct 30 05:05:47.417590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.417562 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-model-cache" pod "59ee8b24-fe74-4b1f-9f63-027e5fa5ec57" (UID: "59ee8b24-fe74-4b1f-9f63-027e5fa5ec57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:47.417776 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.417760 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:47.418210 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.418182 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-home" pod "59ee8b24-fe74-4b1f-9f63-027e5fa5ec57" (UID: "59ee8b24-fe74-4b1f-9f63-027e5fa5ec57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:47.420746 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.420717 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-kube-api-access-mvq8s" pod "59ee8b24-fe74-4b1f-9f63-027e5fa5ec57" (UID: "59ee8b24-fe74-4b1f-9f63-027e5fa5ec57"). InnerVolumeSpecName "kube-api-access-mvq8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:05:47.422600 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.422571 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-dshm" pod "59ee8b24-fe74-4b1f-9f63-027e5fa5ec57" (UID: "59ee8b24-fe74-4b1f-9f63-027e5fa5ec57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:47.422712 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.422635 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tmp-dir" pod "59ee8b24-fe74-4b1f-9f63-027e5fa5ec57" (UID: "59ee8b24-fe74-4b1f-9f63-027e5fa5ec57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:05:47.433597 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.431609 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tls-certs" pod "59ee8b24-fe74-4b1f-9f63-027e5fa5ec57" (UID: "59ee8b24-fe74-4b1f-9f63-027e5fa5ec57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:05:47.433597 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.432757 4793 generic.go:356] "Generic (PLEG): container finished" podID="59ee8b24-fe74-4b1f-9f63-027e5fa5ec57" containerID="8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35" exitCode=0 Oct 30 05:05:47.433597 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.432864 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" event={"ID":"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57","Type":"ContainerDied","Data":"8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35"} Oct 30 05:05:47.433597 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.432900 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" event={"ID":"59ee8b24-fe74-4b1f-9f63-027e5fa5ec57","Type":"ContainerDied","Data":"88e4882c7c5d201ab86fbf7b430927c38d4603dcd6ef2a2f67d3a814065f5bab"} Oct 30 05:05:47.433597 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.432920 4793 scope.go:122] "RemoveContainer" containerID="8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35" Oct 30 05:05:47.433597 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.433063 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7" Oct 30 05:05:47.441308 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.441282 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vq58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:05:47.441407 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.441334 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 05:05:47.480824 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.480797 4793 scope.go:122] "RemoveContainer" containerID="8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35" Oct 30 05:05:47.481433 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:05:47.481383 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35\": container with ID starting with 8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35 not found: ID does not exist" containerID="8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35" Oct 30 05:05:47.481532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.481494 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35"} err="failed to get container status \"8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35\": rpc error: code = NotFound desc = could not find container \"8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35\": container with ID starting with 8c3400d9b504424b799ea2aa700083e567fd07047de5eec9466452f3c31b6c35 not found: ID does not exist" Oct 30 05:05:47.505474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.505449 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7"] Oct 30 05:05:47.510383 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.510357 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7700a7/test-model-7700a7-kserve-6d6cb5b69f-jxgb7"] Oct 30 05:05:47.519508 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.519479 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvq8s\" (UniqueName: \"kubernetes.io/projected/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-kube-api-access-mvq8s\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:47.519508 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.519501 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:47.519645 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.519515 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:47.519645 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.519524 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:47.519645 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:47.519531 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:05:48.441159 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:48.441124 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vq58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:05:48.441331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:48.441181 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 05:05:48.847432 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:48.847343 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ee8b24-fe74-4b1f-9f63-027e5fa5ec57" path="/var/lib/kubelet/pods/59ee8b24-fe74-4b1f-9f63-027e5fa5ec57/volumes" Oct 30 05:05:49.157153 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:49.152168 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:49.375730 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:49.371243 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:49.441099 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:49.441064 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vq58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:05:49.441282 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:49.441117 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 05:05:49.595159 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:49.595115 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:50.441683 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:50.441646 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-vq58x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 05:05:50.442063 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:50.441712 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 05:05:51.762693 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:51.762633 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:51.763301 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:51.763281 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:51.764615 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:51.764591 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:05:51.911277 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:51.911246 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:52.591226 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:52.591192 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:53.951473 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:53.951406 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:54.332528 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:54.332445 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:55.186822 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:55.186709 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:55.400776 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:55.400738 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:56.112717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:56.112684 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:56.329788 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:56.329751 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:57.824571 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:57.824536 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:58.338984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:58.338941 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:58.681893 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:58.681854 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:05:59.629127 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:05:59.629037 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:00.079029 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:00.078982 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:01.300083 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:01.300043 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:02.209777 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:02.209740 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:02.943834 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:02.943791 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:04.631511 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:04.631470 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:05.033395 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:05.033357 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:05.263885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:05.263847 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:06.318955 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:06.318921 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:07.985652 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:07.985616 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:09.710133 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:09.710035 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:11.220905 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:06:11.220871 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:06:11.465472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:11.465431 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:13.106333 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:13.106294 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:14.702205 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:14.702169 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:16.408261 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:16.408218 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:18.137191 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:18.137151 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:19.187354 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:19.187315 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:19.546779 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:19.546607 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:20.233312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:20.231800 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" podUID="e70bca52-d177-4b80-b29b-6e0d64549ef0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:06:20.235001 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:20.234806 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:20.356441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:20.331506 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" podUID="c9e51d14-8150-4f8c-9172-845b3b99eca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:06:20.361991 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:20.361670 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" podUID="439ce07a-ea3d-4b97-a8ed-7655a840c873" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:06:20.402513 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:20.402475 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" podUID="d1176b48-69f0-47e0-9de0-da5d70421762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:06:20.478443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:20.474808 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" podUID="1715329f-b8ff-48a2-8c08-32e93a0b14da" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:06:20.617664 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:20.561496 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" podUID="f53f3359-41ba-40d7-9820-c2409a0a6753" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:06:20.890625 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:20.876181 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:21.239114 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:21.239073 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:22.018142 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:22.018105 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:23.641460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:23.641392 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:25.115000 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:25.114959 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:25.839742 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:25.839710 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:26.805562 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:26.805354 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:27.019339 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:27.019310 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:27.246016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:27.245851 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:27.721041 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:27.720994 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:27.984779 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:27.984674 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:28.202045 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:28.202006 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:28.429938 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:28.429906 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:29.168788 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:29.168750 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:29.501197 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:29.501164 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:29.974001 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:29.973957 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:30.836432 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:30.836376 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:31.326561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:31.326525 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:33.086054 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:33.086019 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:34.006743 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:34.006705 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:34.694275 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:34.694238 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:35.588311 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:35.588271 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:35.804172 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:35.804140 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:36.023450 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:36.022895 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:36.550586 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:36.550546 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:36.894318 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:36.894240 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:38.237852 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:38.237819 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:40.013717 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:40.013676 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:41.220074 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:06:41.220039 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:06:41.825007 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:41.824975 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:42.059824 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:42.059778 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:43.461819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:43.461773 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:45.066361 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:45.066320 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:46.754487 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:46.754088 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:48.541967 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:48.541930 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:50.023136 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:50.023101 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b"] Oct 30 05:06:50.023642 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:50.023292 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" podUID="61cb982d-6448-409d-abdd-a6aa475760fd" containerName="istio-proxy" containerID="cri-o://39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc" gracePeriod=30 Oct 30 05:06:50.097979 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:50.097938 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:51.838799 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:51.838761 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:53.510895 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:53.510859 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:55.106143 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.106097 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:55.152284 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.152246 4793 patch_prober.go:29] interesting pod/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" start-of-body= Oct 30 05:06:55.152493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.152293 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" podUID="61cb982d-6448-409d-abdd-a6aa475760fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.219:15021/healthz/ready\": dial tcp 10.133.0.219:15021: connect: connection refused" Oct 30 05:06:55.318498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.318474 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 05:06:55.348912 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.348884 4793 generic.go:356] "Generic (PLEG): container finished" podID="61cb982d-6448-409d-abdd-a6aa475760fd" containerID="39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc" exitCode=0 Oct 30 05:06:55.349069 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.348924 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" Oct 30 05:06:55.349069 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.348990 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" event={"ID":"61cb982d-6448-409d-abdd-a6aa475760fd","Type":"ContainerDied","Data":"39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc"} Oct 30 05:06:55.349069 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.349042 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b" event={"ID":"61cb982d-6448-409d-abdd-a6aa475760fd","Type":"ContainerDied","Data":"4d01cbf8dc1ff5ffc2890c3ba641cc138532a37cfa864cb6c976b4118b8c31a3"} Oct 30 05:06:55.349069 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.349058 4793 scope.go:122] "RemoveContainer" containerID="39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc" Oct 30 05:06:55.360061 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.360045 4793 scope.go:122] "RemoveContainer" containerID="39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc" Oct 30 05:06:55.360380 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:06:55.360363 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc\": container with ID starting with 39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc not found: ID does not exist" containerID="39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc" Oct 30 05:06:55.360465 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.360387 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc"} err="failed to get container status \"39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc\": rpc error: code = NotFound desc = could not find container \"39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc\": container with ID starting with 39d13f30398d58aee482ae2a861e6dda3ea363990e8b33b6f205e50f1da5b2dc not found: ID does not exist" Oct 30 05:06:55.417130 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417097 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-data\") pod \"61cb982d-6448-409d-abdd-a6aa475760fd\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " Oct 30 05:06:55.417323 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417175 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-certs\") pod \"61cb982d-6448-409d-abdd-a6aa475760fd\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " Oct 30 05:06:55.417323 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417246 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61cb982d-6448-409d-abdd-a6aa475760fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cb982d-6448-409d-abdd-a6aa475760fd-istio-podinfo\") pod \"61cb982d-6448-409d-abdd-a6aa475760fd\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " Oct 30 05:06:55.417323 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417312 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61cb982d-6448-409d-abdd-a6aa475760fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cb982d-6448-409d-abdd-a6aa475760fd-istiod-ca-cert\") pod \"61cb982d-6448-409d-abdd-a6aa475760fd\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " Oct 30 05:06:55.417585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417375 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-data" pod "61cb982d-6448-409d-abdd-a6aa475760fd" (UID: "61cb982d-6448-409d-abdd-a6aa475760fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:06:55.417585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417381 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-credential-socket\") pod \"61cb982d-6448-409d-abdd-a6aa475760fd\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " Oct 30 05:06:55.417585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417510 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-certs" pod "61cb982d-6448-409d-abdd-a6aa475760fd" (UID: "61cb982d-6448-409d-abdd-a6aa475760fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:06:55.417585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417542 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-socket\") pod \"61cb982d-6448-409d-abdd-a6aa475760fd\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " Oct 30 05:06:55.417819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417643 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-kube-api-access-58phc\" (UniqueName: \"kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-kube-api-access-58phc\") pod \"61cb982d-6448-409d-abdd-a6aa475760fd\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " Oct 30 05:06:55.417819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417671 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-credential-socket" pod "61cb982d-6448-409d-abdd-a6aa475760fd" (UID: "61cb982d-6448-409d-abdd-a6aa475760fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:06:55.417819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417790 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-istio-token\" (UniqueName: \"kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-istio-token\") pod \"61cb982d-6448-409d-abdd-a6aa475760fd\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " Oct 30 05:06:55.417988 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417850 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-envoy\") pod \"61cb982d-6448-409d-abdd-a6aa475760fd\" (UID: \"61cb982d-6448-409d-abdd-a6aa475760fd\") " Oct 30 05:06:55.417988 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.417850 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61cb982d-6448-409d-abdd-a6aa475760fd-istiod-ca-cert" pod "61cb982d-6448-409d-abdd-a6aa475760fd" (UID: "61cb982d-6448-409d-abdd-a6aa475760fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:06:55.418699 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.418672 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cb982d-6448-409d-abdd-a6aa475760fd-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:06:55.418852 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.418701 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:06:55.418852 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.418715 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:06:55.418852 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.418730 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:06:55.419048 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.418911 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-socket" pod "61cb982d-6448-409d-abdd-a6aa475760fd" (UID: "61cb982d-6448-409d-abdd-a6aa475760fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:06:55.420112 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.420077 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61cb982d-6448-409d-abdd-a6aa475760fd-istio-podinfo" pod "61cb982d-6448-409d-abdd-a6aa475760fd" (UID: "61cb982d-6448-409d-abdd-a6aa475760fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:06:55.420933 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.420896 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-istio-token" pod "61cb982d-6448-409d-abdd-a6aa475760fd" (UID: "61cb982d-6448-409d-abdd-a6aa475760fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:06:55.420933 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.420905 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-envoy" pod "61cb982d-6448-409d-abdd-a6aa475760fd" (UID: "61cb982d-6448-409d-abdd-a6aa475760fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:06:55.421109 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.420954 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-kube-api-access-58phc" pod "61cb982d-6448-409d-abdd-a6aa475760fd" (UID: "61cb982d-6448-409d-abdd-a6aa475760fd"). InnerVolumeSpecName "kube-api-access-58phc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:06:55.520453 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.520388 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cb982d-6448-409d-abdd-a6aa475760fd-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:06:55.520453 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.520427 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:06:55.520453 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.520438 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58phc\" (UniqueName: \"kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-kube-api-access-58phc\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:06:55.520453 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.520448 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61cb982d-6448-409d-abdd-a6aa475760fd-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:06:55.520453 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.520457 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cb982d-6448-409d-abdd-a6aa475760fd-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:06:55.702487 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.702439 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b"] Oct 30 05:06:55.711246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:55.711220 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-e070ea-openshift-default-54d4b8cc65-prc2b"] Oct 30 05:06:56.832215 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:56.832180 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:56.849208 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:56.849181 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61cb982d-6448-409d-abdd-a6aa475760fd" path="/var/lib/kubelet/pods/61cb982d-6448-409d-abdd-a6aa475760fd/volumes" Oct 30 05:06:58.487741 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:58.487700 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:06:58.702650 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:06:58.702618 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:00.201612 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:00.201570 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:01.563604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:01.563567 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:02.277565 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:02.277526 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:03.962041 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:03.962006 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:05.603008 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:05.602965 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:06.782604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:06.782564 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:07.010587 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:07.010548 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:07.244456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:07.244396 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:07.505178 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:07.505090 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:07.748615 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:07.748581 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f"] Oct 30 05:07:07.748867 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:07.748812 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" containerID="cri-o://6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41" gracePeriod=30 Oct 30 05:07:07.918518 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:07.917953 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:08.776873 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:08.776840 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:09.148210 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:09.148110 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:10.787148 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:10.787112 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:11.220809 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:07:11.220778 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:07:12.521244 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:12.521199 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:13.058955 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.058927 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 05:07:13.196108 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.196074 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-data\") pod \"65430b4d-69e5-4e30-8a47-9c1e9003759c\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " Oct 30 05:07:13.196312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.196159 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-certs\") pod \"65430b4d-69e5-4e30-8a47-9c1e9003759c\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " Oct 30 05:07:13.196312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.196199 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-podinfo\") pod \"65430b4d-69e5-4e30-8a47-9c1e9003759c\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " Oct 30 05:07:13.196312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.196266 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-credential-socket\") pod \"65430b4d-69e5-4e30-8a47-9c1e9003759c\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " Oct 30 05:07:13.196312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.196296 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-socket\") pod \"65430b4d-69e5-4e30-8a47-9c1e9003759c\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " Oct 30 05:07:13.196577 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.196333 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65430b4d-69e5-4e30-8a47-9c1e9003759c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65430b4d-69e5-4e30-8a47-9c1e9003759c-istiod-ca-cert\") pod \"65430b4d-69e5-4e30-8a47-9c1e9003759c\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " Oct 30 05:07:13.196577 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.196377 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-data" pod "65430b4d-69e5-4e30-8a47-9c1e9003759c" (UID: "65430b4d-69e5-4e30-8a47-9c1e9003759c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:07:13.196577 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.196428 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-envoy\") pod \"65430b4d-69e5-4e30-8a47-9c1e9003759c\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " Oct 30 05:07:13.196748 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.196646 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-credential-socket" pod "65430b4d-69e5-4e30-8a47-9c1e9003759c" (UID: "65430b4d-69e5-4e30-8a47-9c1e9003759c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:07:13.196748 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.196648 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-token\" (UniqueName: \"kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-token\") pod \"65430b4d-69e5-4e30-8a47-9c1e9003759c\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " Oct 30 05:07:13.196848 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.196759 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-kube-api-access-7rspq\" (UniqueName: \"kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-kube-api-access-7rspq\") pod \"65430b4d-69e5-4e30-8a47-9c1e9003759c\" (UID: \"65430b4d-69e5-4e30-8a47-9c1e9003759c\") " Oct 30 05:07:13.197286 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.197258 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65430b4d-69e5-4e30-8a47-9c1e9003759c-istiod-ca-cert" pod "65430b4d-69e5-4e30-8a47-9c1e9003759c" (UID: "65430b4d-69e5-4e30-8a47-9c1e9003759c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:07:13.197864 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.197574 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:13.197864 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.197597 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:13.197864 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.197611 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65430b4d-69e5-4e30-8a47-9c1e9003759c-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:13.198095 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.198058 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-certs" pod "65430b4d-69e5-4e30-8a47-9c1e9003759c" (UID: "65430b4d-69e5-4e30-8a47-9c1e9003759c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:07:13.198251 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.198227 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-socket" pod "65430b4d-69e5-4e30-8a47-9c1e9003759c" (UID: "65430b4d-69e5-4e30-8a47-9c1e9003759c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:07:13.199454 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.199392 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-token" pod "65430b4d-69e5-4e30-8a47-9c1e9003759c" (UID: "65430b4d-69e5-4e30-8a47-9c1e9003759c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:07:13.199986 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.199966 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-podinfo" pod "65430b4d-69e5-4e30-8a47-9c1e9003759c" (UID: "65430b4d-69e5-4e30-8a47-9c1e9003759c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:07:13.200068 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.200022 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-envoy" pod "65430b4d-69e5-4e30-8a47-9c1e9003759c" (UID: "65430b4d-69e5-4e30-8a47-9c1e9003759c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:07:13.202218 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.202192 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-kube-api-access-7rspq" pod "65430b4d-69e5-4e30-8a47-9c1e9003759c" (UID: "65430b4d-69e5-4e30-8a47-9c1e9003759c"). InnerVolumeSpecName "kube-api-access-7rspq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:07:13.299447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.299361 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:13.299447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.299388 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rspq\" (UniqueName: \"kubernetes.io/projected/65430b4d-69e5-4e30-8a47-9c1e9003759c-kube-api-access-7rspq\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:13.299447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.299398 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:13.299447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.299408 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:13.299447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.299452 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:13.299714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.299462 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65430b4d-69e5-4e30-8a47-9c1e9003759c-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:13.509791 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.509758 4793 generic.go:356] "Generic (PLEG): container finished" podID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerID="6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41" exitCode=0 Oct 30 05:07:13.509997 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.509806 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" Oct 30 05:07:13.509997 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.509854 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" event={"ID":"65430b4d-69e5-4e30-8a47-9c1e9003759c","Type":"ContainerDied","Data":"6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41"} Oct 30 05:07:13.509997 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.509912 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f" event={"ID":"65430b4d-69e5-4e30-8a47-9c1e9003759c","Type":"ContainerDied","Data":"789438c125c67cf14fdf439da33a90985271765ee7732894772978ec6b1fc5c3"} Oct 30 05:07:13.509997 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.509942 4793 scope.go:122] "RemoveContainer" containerID="6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41" Oct 30 05:07:13.526142 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.526123 4793 scope.go:122] "RemoveContainer" containerID="70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7" Oct 30 05:07:13.536234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.536218 4793 scope.go:122] "RemoveContainer" containerID="6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41" Oct 30 05:07:13.536528 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:07:13.536508 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41\": container with ID starting with 6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41 not found: ID does not exist" containerID="6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41" Oct 30 05:07:13.536585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.536539 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41"} err="failed to get container status \"6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41\": rpc error: code = NotFound desc = could not find container \"6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41\": container with ID starting with 6183f03d8ca9d7fcb6e670a9a7029e8495567158e11e943d78c23cb40c38fc41 not found: ID does not exist" Oct 30 05:07:13.536585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.536560 4793 scope.go:122] "RemoveContainer" containerID="70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7" Oct 30 05:07:13.536822 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:07:13.536802 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7\": container with ID starting with 70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7 not found: ID does not exist" containerID="70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7" Oct 30 05:07:13.536865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.536830 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7"} err="failed to get container status \"70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7\": rpc error: code = NotFound desc = could not find container \"70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7\": container with ID starting with 70433e8161bf0e5b54c16360a70b23c7ef0a034d5eb973d7b0251f558e7f66b7 not found: ID does not exist" Oct 30 05:07:13.563308 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.563246 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f"] Oct 30 05:07:13.575372 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:13.575348 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9f158a-openshift-default-794665b7cf-74d6f"] Oct 30 05:07:14.133501 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:14.133444 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:14.418647 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:14.418565 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9"] Oct 30 05:07:14.645450 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:14.643205 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:14.848241 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:14.848209 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" path="/var/lib/kubelet/pods/65430b4d-69e5-4e30-8a47-9c1e9003759c/volumes" Oct 30 05:07:14.863072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:14.863023 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:15.523235 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:15.523197 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:15.907499 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:15.907368 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:17.360263 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:17.360222 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:18.852116 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:18.852075 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:19.419391 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:19.419348 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" podUID="357e05d0-d457-4cb2-8d86-4a7990e6d6eb" containerName="payload-pre-processing" containerID="cri-o://fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96" gracePeriod=25 Oct 30 05:07:19.820388 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:19.820350 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:20.357559 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:20.357520 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" podUID="357e05d0-d457-4cb2-8d86-4a7990e6d6eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:07:21.278661 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:21.278623 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:21.595504 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:21.595377 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:21.807498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:21.807462 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:22.261521 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:22.261480 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:22.668492 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:22.662879 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:23.360938 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:23.360895 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:23.796384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:23.796351 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:24.125131 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.125047 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:24.127247 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.127221 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl"] Oct 30 05:07:24.128198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128175 4793 cpu_manager.go:414] "containerMap: removing container" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" Oct 30 05:07:24.128198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128197 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" Oct 30 05:07:24.128472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128224 4793 cpu_manager.go:414] "containerMap: removing container" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" Oct 30 05:07:24.128472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128231 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" Oct 30 05:07:24.128472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128256 4793 cpu_manager.go:414] "containerMap: removing container" podUID="61cb982d-6448-409d-abdd-a6aa475760fd" containerName="istio-proxy" Oct 30 05:07:24.128472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128262 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61cb982d-6448-409d-abdd-a6aa475760fd" containerName="istio-proxy" Oct 30 05:07:24.128472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128289 4793 cpu_manager.go:414] "containerMap: removing container" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" Oct 30 05:07:24.128472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128295 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b6d0eb-2e34-43b0-9b83-c448f4b0fc28" containerName="istio-proxy" Oct 30 05:07:24.128472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128311 4793 cpu_manager.go:414] "containerMap: removing container" podUID="59ee8b24-fe74-4b1f-9f63-027e5fa5ec57" containerName="main" Oct 30 05:07:24.128472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128318 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ee8b24-fe74-4b1f-9f63-027e5fa5ec57" containerName="main" Oct 30 05:07:24.128868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128534 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" Oct 30 05:07:24.128868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128554 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="61cb982d-6448-409d-abdd-a6aa475760fd" containerName="istio-proxy" Oct 30 05:07:24.128868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.128573 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ee8b24-fe74-4b1f-9f63-027e5fa5ec57" containerName="main" Oct 30 05:07:24.149348 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.148225 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.155934 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.155910 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl"] Oct 30 05:07:24.163094 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.163072 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b5496290-openshift-default-dockercfg-c7dbk\"" Oct 30 05:07:24.221691 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.221643 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-credential-socket\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.221909 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.221708 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfn9z\" (UniqueName: \"kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-kube-api-access-bfn9z\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.221909 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.221860 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istiod-ca-cert\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.222047 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.221970 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-data\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.222146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.222116 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-envoy\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.222201 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.222182 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-socket\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.222312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.222295 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-token\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.222475 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.222434 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-podinfo\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.222551 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.222487 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-certs\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.324686 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.324634 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-credential-socket\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.324969 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.324946 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfn9z\" (UniqueName: \"kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-kube-api-access-bfn9z\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.325114 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.325099 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istiod-ca-cert\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.325253 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.325238 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-data\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.325392 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.325380 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-envoy\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.325531 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.325517 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-socket\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.325673 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.325659 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-token\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.325850 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.325837 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-podinfo\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.325944 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.325931 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-certs\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.326578 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.326556 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-certs\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.326961 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.326943 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-credential-socket\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.328060 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.328034 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istiod-ca-cert\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.328550 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.328527 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-data\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.346498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.342855 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-socket\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.348483 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.348364 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:24.366283 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.366222 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-podinfo\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.366672 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.366653 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-envoy\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.369247 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.369203 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfn9z\" (UniqueName: \"kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-kube-api-access-bfn9z\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.374048 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.374025 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-token\") pod \"e2e-mt-b5496290-openshift-default-786dddd54f-tplpl\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.474789 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.474749 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:24.572084 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.572046 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:24.811359 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.811328 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl"] Oct 30 05:07:24.811753 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:07:24.811642 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943d71d5_69a1_45ba_8cb1_0532e9d15b42.slice/crio-46d21623613361cb9f62c8ff7313411dec93c2aecd1a389e856093ecee819beb WatchSource:0}: Error finding container 46d21623613361cb9f62c8ff7313411dec93c2aecd1a389e856093ecee819beb: Status 404 returned error can't find the container with id 46d21623613361cb9f62c8ff7313411dec93c2aecd1a389e856093ecee819beb Oct 30 05:07:24.814637 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.814602 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:07:24.814780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.814682 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:07:24.814780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:24.814738 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:07:25.195188 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:25.195141 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:25.610684 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:25.610605 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" event={"ID":"943d71d5-69a1-45ba-8cb1-0532e9d15b42","Type":"ContainerStarted","Data":"b6d73d7858ebc0dbabcd5194bb65350ed784605831a1d16ca998fbcaf9be87c7"} Oct 30 05:07:25.610684 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:25.610662 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" event={"ID":"943d71d5-69a1-45ba-8cb1-0532e9d15b42","Type":"ContainerStarted","Data":"46d21623613361cb9f62c8ff7313411dec93c2aecd1a389e856093ecee819beb"} Oct 30 05:07:25.666481 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:25.666402 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" podStartSLOduration=1.66638616 podStartE2EDuration="1.66638616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:07:25.66563877 +0000 UTC m=+4168.951513543" watchObservedRunningTime="2026-10-30 05:07:25.66638616 +0000 UTC m=+4168.952260936" Oct 30 05:07:26.475829 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:26.475793 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:26.483319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:26.483289 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:26.617844 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:26.617815 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:26.618668 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:26.618646 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:07:26.887536 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:26.887433 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:28.174983 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:28.174944 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:28.475128 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:28.475083 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:28.699731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:28.699698 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:29.853056 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:29.853019 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:30.100559 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:30.100519 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:30.358142 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:30.358097 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" podUID="357e05d0-d457-4cb2-8d86-4a7990e6d6eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:07:30.622488 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:30.622372 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:31.695915 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:31.695876 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:32.014377 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:32.014282 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:32.319111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:32.319024 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:33.575044 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:33.575011 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:34.106020 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:34.105979 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:35.714628 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:35.714587 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:37.398853 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:37.398819 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:39.049613 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.049575 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:39.699097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.699068 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" Oct 30 05:07:39.734907 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.734879 4793 generic.go:356] "Generic (PLEG): container finished" podID="357e05d0-d457-4cb2-8d86-4a7990e6d6eb" containerID="fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96" exitCode=0 Oct 30 05:07:39.735046 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.734918 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" Oct 30 05:07:39.735046 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.734961 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" event={"ID":"357e05d0-d457-4cb2-8d86-4a7990e6d6eb","Type":"ContainerDied","Data":"fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96"} Oct 30 05:07:39.735046 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.735014 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9" event={"ID":"357e05d0-d457-4cb2-8d86-4a7990e6d6eb","Type":"ContainerDied","Data":"e49b0aeae3694b9ea797016e7c0cb314642985fc5d50fd1fedb9e6ea75a95594"} Oct 30 05:07:39.735046 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.735042 4793 scope.go:122] "RemoveContainer" containerID="fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96" Oct 30 05:07:39.747569 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.747538 4793 scope.go:122] "RemoveContainer" containerID="fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96" Oct 30 05:07:39.747906 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:07:39.747876 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96\": container with ID starting with fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96 not found: ID does not exist" containerID="fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96" Oct 30 05:07:39.747906 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.747915 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96"} err="failed to get container status \"fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96\": rpc error: code = NotFound desc = could not find container \"fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96\": container with ID starting with fce214f3e50d44b9fc7dcfea85c3cc74a646d2cfaf641c8c540474a8b6147d96 not found: ID does not exist" Oct 30 05:07:39.860460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.860394 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-kube-api-access-sntlg\" (UniqueName: \"kubernetes.io/projected/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-kube-api-access-sntlg\") pod \"357e05d0-d457-4cb2-8d86-4a7990e6d6eb\" (UID: \"357e05d0-d457-4cb2-8d86-4a7990e6d6eb\") " Oct 30 05:07:39.860645 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.860622 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-plugins-config-volume\") pod \"357e05d0-d457-4cb2-8d86-4a7990e6d6eb\" (UID: \"357e05d0-d457-4cb2-8d86-4a7990e6d6eb\") " Oct 30 05:07:39.861062 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.861037 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-plugins-config-volume" pod "357e05d0-d457-4cb2-8d86-4a7990e6d6eb" (UID: "357e05d0-d457-4cb2-8d86-4a7990e6d6eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:07:39.862534 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.862505 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-kube-api-access-sntlg" pod "357e05d0-d457-4cb2-8d86-4a7990e6d6eb" (UID: "357e05d0-d457-4cb2-8d86-4a7990e6d6eb"). InnerVolumeSpecName "kube-api-access-sntlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:07:39.880326 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.880288 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:39.964445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.964350 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:39.964445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:39.964380 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sntlg\" (UniqueName: \"kubernetes.io/projected/357e05d0-d457-4cb2-8d86-4a7990e6d6eb-kube-api-access-sntlg\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:40.083529 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:40.083495 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9"] Oct 30 05:07:40.093848 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:40.093817 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:40.095143 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:40.095120 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-26b72f-5d5bd5d9f-x42j9"] Oct 30 05:07:40.642350 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:40.642309 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:40.848149 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:40.848117 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357e05d0-d457-4cb2-8d86-4a7990e6d6eb" path="/var/lib/kubelet/pods/357e05d0-d457-4cb2-8d86-4a7990e6d6eb/volumes" Oct 30 05:07:41.221146 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:07:41.221114 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:07:42.308073 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:42.308035 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:44.101894 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:44.101854 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:45.822444 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:45.816848 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:46.763996 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:46.763950 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:47.468743 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:47.468703 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:49.119312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:49.119267 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:50.335731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.335694 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc"] Oct 30 05:07:50.336236 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.335977 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" podUID="2996a1cc-21e1-4025-bb05-a52f82f82958" containerName="maas-api" containerID="cri-o://1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430" gracePeriod=30 Oct 30 05:07:50.505277 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.505240 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:50.745474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.732286 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:07:50.876443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.875146 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2996a1cc-21e1-4025-bb05-a52f82f82958-kube-api-access-4rb4z\" (UniqueName: \"kubernetes.io/projected/2996a1cc-21e1-4025-bb05-a52f82f82958-kube-api-access-4rb4z\") pod \"2996a1cc-21e1-4025-bb05-a52f82f82958\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " Oct 30 05:07:50.876443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.875219 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-metrics-certs\") pod \"2996a1cc-21e1-4025-bb05-a52f82f82958\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " Oct 30 05:07:50.876443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.875308 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2996a1cc-21e1-4025-bb05-a52f82f82958-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2996a1cc-21e1-4025-bb05-a52f82f82958-openshift-service-ca\") pod \"2996a1cc-21e1-4025-bb05-a52f82f82958\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " Oct 30 05:07:50.876443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.875493 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-maas-api-tls\") pod \"2996a1cc-21e1-4025-bb05-a52f82f82958\" (UID: \"2996a1cc-21e1-4025-bb05-a52f82f82958\") " Oct 30 05:07:50.878572 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.876927 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2996a1cc-21e1-4025-bb05-a52f82f82958-openshift-service-ca" pod "2996a1cc-21e1-4025-bb05-a52f82f82958" (UID: "2996a1cc-21e1-4025-bb05-a52f82f82958"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:07:50.889782 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.889747 4793 generic.go:356] "Generic (PLEG): container finished" podID="2996a1cc-21e1-4025-bb05-a52f82f82958" containerID="1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430" exitCode=0 Oct 30 05:07:50.889898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.889749 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-metrics-certs" pod "2996a1cc-21e1-4025-bb05-a52f82f82958" (UID: "2996a1cc-21e1-4025-bb05-a52f82f82958"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:07:50.889898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.889849 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" Oct 30 05:07:50.891487 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.891455 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-maas-api-tls" pod "2996a1cc-21e1-4025-bb05-a52f82f82958" (UID: "2996a1cc-21e1-4025-bb05-a52f82f82958"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:07:50.891588 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.891500 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2996a1cc-21e1-4025-bb05-a52f82f82958-kube-api-access-4rb4z" pod "2996a1cc-21e1-4025-bb05-a52f82f82958" (UID: "2996a1cc-21e1-4025-bb05-a52f82f82958"). InnerVolumeSpecName "kube-api-access-4rb4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:07:50.894210 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.894183 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" event={"ID":"2996a1cc-21e1-4025-bb05-a52f82f82958","Type":"ContainerDied","Data":"1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430"} Oct 30 05:07:50.894312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.894245 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc" event={"ID":"2996a1cc-21e1-4025-bb05-a52f82f82958","Type":"ContainerDied","Data":"070bb629916ae23f5f58f0cd3b392f005c66b080fe48c576eb3c20669d2e472a"} Oct 30 05:07:50.894312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.894279 4793 scope.go:122] "RemoveContainer" containerID="1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430" Oct 30 05:07:50.906337 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.906320 4793 scope.go:122] "RemoveContainer" containerID="1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430" Oct 30 05:07:50.906688 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:07:50.906666 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430\": container with ID starting with 1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430 not found: ID does not exist" containerID="1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430" Oct 30 05:07:50.906774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.906698 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430"} err="failed to get container status \"1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430\": rpc error: code = NotFound desc = could not find container \"1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430\": container with ID starting with 1ebbdf1b66ef5a3a9ce69732aa06d7dcf82fbcdb8ebb930a3f7af1e16fce1430 not found: ID does not exist" Oct 30 05:07:50.978005 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.977953 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rb4z\" (UniqueName: \"kubernetes.io/projected/2996a1cc-21e1-4025-bb05-a52f82f82958-kube-api-access-4rb4z\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:50.978005 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.977989 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:50.978005 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.978004 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2996a1cc-21e1-4025-bb05-a52f82f82958-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:50.978330 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:50.978018 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2996a1cc-21e1-4025-bb05-a52f82f82958-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:07:51.218731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:51.218698 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:51.236444 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:51.236354 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc"] Oct 30 05:07:51.241604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:51.241584 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-26b72f-7b4899b6c8-nplkc"] Oct 30 05:07:52.828119 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:52.828083 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:52.849233 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:52.849202 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2996a1cc-21e1-4025-bb05-a52f82f82958" path="/var/lib/kubelet/pods/2996a1cc-21e1-4025-bb05-a52f82f82958/volumes" Oct 30 05:07:54.517699 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:54.517664 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:54.964641 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:54.964606 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:55.188867 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:55.188825 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:55.411807 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:55.411713 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:55.684122 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:55.684087 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:56.151451 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:56.151330 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:56.610603 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:56.610567 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:58.416319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:58.416286 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:07:58.648851 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:07:58.648814 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:00.058855 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:00.058819 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:00.756233 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:00.756197 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:00.965008 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:00.964972 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:01.710867 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:01.710833 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:03.430801 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:03.430764 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:05.106192 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:05.106149 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:06.863299 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:06.863255 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:08.489691 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:08.489658 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:08.699003 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:08.698965 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:10.214001 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:10.213909 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:11.220945 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:08:11.220889 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:08:11.991234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:11.991193 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:14.026782 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:14.026737 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:15.718498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:15.718454 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:17.422684 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:17.422651 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:19.009855 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:19.009818 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:19.774532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:19.774482 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:20.735066 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:20.735013 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:22.415584 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:22.415542 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:24.069189 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:24.069151 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:25.271524 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:25.271462 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:25.483795 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:25.483754 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:25.777955 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:25.777918 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:27.403995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:27.403482 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:29.054265 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:29.054229 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:30.816722 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:30.816684 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:32.504085 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:32.504042 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:34.377172 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:34.377016 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:34.992350 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:34.992310 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:36.203313 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:36.203277 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:37.800656 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:37.800618 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:39.425463 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:39.425176 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:39.648097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:39.648063 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:41.044133 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:41.044092 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:41.220978 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:08:41.220934 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:08:42.660080 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:42.659597 4793 patch_prober.go:29] interesting pod/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:08:42.660080 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:42.659688 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:08:42.743622 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:42.743582 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:44.376769 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:44.376728 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:45.471604 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:45.471567 4793 generic.go:356] "Generic (PLEG): container finished" podID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerID="b6d73d7858ebc0dbabcd5194bb65350ed784605831a1d16ca998fbcaf9be87c7" exitCode=0 Oct 30 05:08:45.472029 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:45.471632 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" event={"ID":"943d71d5-69a1-45ba-8cb1-0532e9d15b42","Type":"ContainerDied","Data":"b6d73d7858ebc0dbabcd5194bb65350ed784605831a1d16ca998fbcaf9be87c7"} Oct 30 05:08:45.472074 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:45.472043 4793 scope.go:122] "RemoveContainer" containerID="b6d73d7858ebc0dbabcd5194bb65350ed784605831a1d16ca998fbcaf9be87c7" Oct 30 05:08:45.473935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:45.473904 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:08:45.474024 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:45.474000 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:08:45.474076 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:45.474043 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:08:46.271439 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:46.270577 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:46.504988 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:46.504947 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" event={"ID":"943d71d5-69a1-45ba-8cb1-0532e9d15b42","Type":"ContainerStarted","Data":"d8fd3e6a7d41662c51e92677cd0aa092c8210a4ede48f18077bf8dee6fde9bee"} Oct 30 05:08:46.512437 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:46.512392 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:46.730164 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:46.730123 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:47.481784 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:47.481744 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:08:47.482162 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:47.482078 4793 patch_prober.go:29] interesting pod/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Oct 30 05:08:47.482333 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:47.482309 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Oct 30 05:08:48.148043 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:48.148014 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:48.477399 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:48.476920 4793 patch_prober.go:29] interesting pod/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Oct 30 05:08:48.477399 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:48.476975 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Oct 30 05:08:49.483561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:49.483520 4793 patch_prober.go:29] interesting pod/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Oct 30 05:08:49.484169 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:49.484132 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Oct 30 05:08:50.170973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:50.170934 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:50.478829 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:50.478783 4793 patch_prober.go:29] interesting pod/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Oct 30 05:08:50.479035 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:50.478877 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Oct 30 05:08:51.476400 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:51.476355 4793 patch_prober.go:29] interesting pod/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" start-of-body= Oct 30 05:08:51.476913 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:51.476454 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.251:15021/healthz/ready\": dial tcp 10.133.0.251:15021: connect: connection refused" Oct 30 05:08:52.009034 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:52.008993 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:52.480959 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:52.480930 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:08:52.481474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:52.481452 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:08:52.482272 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:52.482250 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:08:53.244571 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:53.244534 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:53.709801 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:53.709758 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:55.185672 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:55.185631 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:55.402321 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:55.402278 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:56.119917 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:56.119865 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:56.337758 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:56.337724 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:56.662698 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:56.662654 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:57.209559 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:57.209518 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:58.727954 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:58.727917 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:08:58.939031 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:08:58.938994 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:00.585440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:00.585386 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:02.199897 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:02.199859 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:03.355786 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:03.355751 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:03.961862 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:03.961816 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:05.715532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:05.715337 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:06.766786 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:06.766744 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:07.321978 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:07.321940 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:08.959735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:08.959699 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:10.358180 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:10.358087 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:10.582776 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:10.582727 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:10.809045 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:10.808992 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:11.221572 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:09:11.221534 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:09:12.428521 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:12.428484 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:14.082860 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:14.082826 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:15.455018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:15.454983 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:16.101456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:16.101379 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:17.431232 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:17.431192 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:17.673447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:17.673246 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:17.907181 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:17.907100 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:18.230363 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:18.230328 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:18.458984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:18.458940 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:18.897965 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:18.897932 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:19.126493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:19.126460 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz"] Oct 30 05:09:19.126981 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:19.126943 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" podUID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" containerName="istio-proxy" containerID="cri-o://65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e" gracePeriod=30 Oct 30 05:09:19.280439 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:19.274124 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:19.735403 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:19.735364 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:20.156102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:20.156002 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:21.362256 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:21.362214 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:21.468671 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:21.468635 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:09:21.468838 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:21.468709 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" podUID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:09:23.037790 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:23.037754 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:24.435403 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.435375 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 05:09:24.498740 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.498651 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:24.605849 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.605816 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-certs\") pod \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " Oct 30 05:09:24.606061 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.605862 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-envoy\") pod \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " Oct 30 05:09:24.606061 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.605902 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-socket\") pod \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " Oct 30 05:09:24.606061 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.606035 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-podinfo\") pod \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " Oct 30 05:09:24.606061 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.606058 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-kube-api-access-vjdrp\" (UniqueName: \"kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-kube-api-access-vjdrp\") pod \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " Oct 30 05:09:24.606306 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.606096 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istiod-ca-cert\") pod \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " Oct 30 05:09:24.606306 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.606156 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-certs" pod "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" (UID: "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:24.606306 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.606272 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-data\") pod \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " Oct 30 05:09:24.606485 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.606306 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-socket" pod "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" (UID: "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:24.606485 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.606326 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-token\" (UniqueName: \"kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-token\") pod \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " Oct 30 05:09:24.606485 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.606377 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-credential-socket\") pod \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\" (UID: \"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2\") " Oct 30 05:09:24.606672 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.606549 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-data" pod "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" (UID: "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:24.606672 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.606631 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istiod-ca-cert" pod "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" (UID: "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:09:24.606672 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.606642 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-credential-socket" pod "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" (UID: "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:24.607124 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.607104 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:09:24.607219 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.607127 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:09:24.607219 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.607140 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:09:24.607219 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.607153 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:09:24.607219 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.607167 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:09:24.608660 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.608637 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-podinfo" pod "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" (UID: "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:09:24.608660 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.608644 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-envoy" pod "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" (UID: "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:09:24.608893 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.608871 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-token" pod "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" (UID: "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:24.608968 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.608937 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-kube-api-access-vjdrp" pod "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" (UID: "72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2"). InnerVolumeSpecName "kube-api-access-vjdrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:09:24.709037 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.709010 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:09:24.709153 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.709043 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjdrp\" (UniqueName: \"kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-kube-api-access-vjdrp\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:09:24.709153 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.709058 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:09:24.709153 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.709073 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:09:24.782997 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.782961 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:24.913440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.908292 4793 generic.go:356] "Generic (PLEG): container finished" podID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" containerID="65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e" exitCode=0 Oct 30 05:09:24.913440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.908492 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" event={"ID":"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2","Type":"ContainerDied","Data":"65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e"} Oct 30 05:09:24.913440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.908545 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" event={"ID":"72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2","Type":"ContainerDied","Data":"a6205e176bd10f7f0c838e2d0d9bf5264f5eb84e8e6130106ef00714f665fccd"} Oct 30 05:09:24.913440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.908579 4793 scope.go:122] "RemoveContainer" containerID="65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e" Oct 30 05:09:24.913440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.908749 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz" Oct 30 05:09:24.990696 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.986670 4793 scope.go:122] "RemoveContainer" containerID="65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e" Oct 30 05:09:24.990696 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:09:24.987343 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e\": container with ID starting with 65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e not found: ID does not exist" containerID="65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e" Oct 30 05:09:24.990696 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:24.987389 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e"} err="failed to get container status \"65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e\": rpc error: code = NotFound desc = could not find container \"65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e\": container with ID starting with 65f5853bc85013000cf29ca3d6c5d34ac91c2e9eb074b635f07e92d8f0fdbc1e not found: ID does not exist" Oct 30 05:09:25.011010 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:25.004983 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:25.011010 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:25.005160 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz"] Oct 30 05:09:25.016685 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:25.016651 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ed8266-openshift-default-9c5b6fdc7-kgjxz"] Oct 30 05:09:26.443544 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:26.443500 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:26.655235 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:26.655196 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:26.920804 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:26.920722 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" path="/var/lib/kubelet/pods/72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2/volumes" Oct 30 05:09:27.027192 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:27.027152 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:28.314874 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:28.314834 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:29.975188 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:29.975156 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:30.187457 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:30.187395 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:31.613566 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:31.613533 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:32.830117 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:32.830031 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:33.346148 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:33.346119 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:34.916686 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:34.916648 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:36.703332 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:36.703284 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:38.293656 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:38.293611 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:40.016405 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:40.016371 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:41.221029 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:09:41.220995 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:09:41.723136 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:41.723102 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:43.368141 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:43.368099 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:45.039339 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:45.039296 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:46.707167 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:46.707126 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:48.403580 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:48.403529 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:50.184987 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:50.184951 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:51.796918 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:51.796884 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:53.515858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:53.515823 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:55.224978 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:55.224937 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:57.030401 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:57.030364 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:58.806012 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:58.805971 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:09:59.024930 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:09:59.024896 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:00.277447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:00.277394 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:00.909615 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:00.909582 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:02.626945 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:02.626907 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:03.843750 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:03.843716 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782"] Oct 30 05:10:03.844317 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:03.843959 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" podUID="3049a33a-cac2-4174-82d4-7e0a7b96fdb0" containerName="maas-api" containerID="cri-o://db300770b387802d795c0985eaee803570a8b259fb6c7e89ca0ad8e05569bc44" gracePeriod=30 Oct 30 05:10:03.996195 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:03.996152 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:04.247467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.243370 4793 generic.go:356] "Generic (PLEG): container finished" podID="3049a33a-cac2-4174-82d4-7e0a7b96fdb0" containerID="db300770b387802d795c0985eaee803570a8b259fb6c7e89ca0ad8e05569bc44" exitCode=0 Oct 30 05:10:04.247467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.243500 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" event={"ID":"3049a33a-cac2-4174-82d4-7e0a7b96fdb0","Type":"ContainerDied","Data":"db300770b387802d795c0985eaee803570a8b259fb6c7e89ca0ad8e05569bc44"} Oct 30 05:10:04.247467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.243557 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" event={"ID":"3049a33a-cac2-4174-82d4-7e0a7b96fdb0","Type":"ContainerDied","Data":"5fa3ab84918727e4e907f1e5dbcdd24730b5a518a4331ae4e86323639e1d615b"} Oct 30 05:10:04.247467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.243582 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fa3ab84918727e4e907f1e5dbcdd24730b5a518a4331ae4e86323639e1d615b" Oct 30 05:10:04.255452 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.253570 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 05:10:04.306442 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.305545 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-maas-api-tls\") pod \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " Oct 30 05:10:04.307455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.306675 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-kube-api-access-6wc4k\" (UniqueName: \"kubernetes.io/projected/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-kube-api-access-6wc4k\") pod \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " Oct 30 05:10:04.307455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.306792 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-metrics-certs\") pod \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " Oct 30 05:10:04.343440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.332531 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-maas-api-tls" pod "3049a33a-cac2-4174-82d4-7e0a7b96fdb0" (UID: "3049a33a-cac2-4174-82d4-7e0a7b96fdb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:10:04.350438 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.343652 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-kube-api-access-6wc4k" pod "3049a33a-cac2-4174-82d4-7e0a7b96fdb0" (UID: "3049a33a-cac2-4174-82d4-7e0a7b96fdb0"). InnerVolumeSpecName "kube-api-access-6wc4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:04.357436 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.355537 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-metrics-certs" pod "3049a33a-cac2-4174-82d4-7e0a7b96fdb0" (UID: "3049a33a-cac2-4174-82d4-7e0a7b96fdb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:10:04.357436 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.355984 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-openshift-service-ca\") pod \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\" (UID: \"3049a33a-cac2-4174-82d4-7e0a7b96fdb0\") " Oct 30 05:10:04.357436 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.356852 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wc4k\" (UniqueName: \"kubernetes.io/projected/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-kube-api-access-6wc4k\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:04.357436 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.356879 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:04.357436 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.356890 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:04.381439 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.380650 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-openshift-service-ca" pod "3049a33a-cac2-4174-82d4-7e0a7b96fdb0" (UID: "3049a33a-cac2-4174-82d4-7e0a7b96fdb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:04.458546 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.458506 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3049a33a-cac2-4174-82d4-7e0a7b96fdb0-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:04.677998 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:04.677959 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:05.250074 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:05.250042 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782" Oct 30 05:10:05.314963 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:05.314933 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782"] Oct 30 05:10:05.329854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:05.329822 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-483bea-6f457495fd-64782"] Oct 30 05:10:06.325046 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:06.325006 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:06.849408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:06.849376 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3049a33a-cac2-4174-82d4-7e0a7b96fdb0" path="/var/lib/kubelet/pods/3049a33a-cac2-4174-82d4-7e0a7b96fdb0/volumes" Oct 30 05:10:06.869499 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:06.869470 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:07.225924 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:07.225886 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:07.763805 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:07.763764 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:07.997753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:07.997720 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml"] Oct 30 05:10:07.998025 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:07.997992 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" containerID="cri-o://815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682" gracePeriod=30 Oct 30 05:10:08.144255 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:08.144163 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:08.651734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:08.651698 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:09.019165 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:09.019127 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:09.341943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:09.341847 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:09.586803 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:09.586770 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:10.168485 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:10.168410 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:10.355312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:10.355279 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl"] Oct 30 05:10:10.355804 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:10.355772 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerName="istio-proxy" containerID="cri-o://5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b" gracePeriod=30 Oct 30 05:10:10.550994 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:10.550953 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:10.793099 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:10.793055 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:11.219401 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:10:11.219368 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:10:11.282798 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:11.282761 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:12.416130 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:12.416087 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:13.293673 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.293644 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 05:10:13.362360 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.362324 4793 generic.go:356] "Generic (PLEG): container finished" podID="07de9748-a77d-4181-8202-6edb393a4882" containerID="815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682" exitCode=0 Oct 30 05:10:13.362554 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.362380 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" Oct 30 05:10:13.362554 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.362409 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" event={"ID":"07de9748-a77d-4181-8202-6edb393a4882","Type":"ContainerDied","Data":"815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682"} Oct 30 05:10:13.362554 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.362542 4793 scope.go:122] "RemoveContainer" containerID="815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682" Oct 30 05:10:13.362738 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.362701 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml" event={"ID":"07de9748-a77d-4181-8202-6edb393a4882","Type":"ContainerDied","Data":"d5c77dc7bc6c5a266c1d249405e7fe396e84216108d79deb08bf70585b83c413"} Oct 30 05:10:13.385494 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.385472 4793 scope.go:122] "RemoveContainer" containerID="ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba" Oct 30 05:10:13.408884 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.408856 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07de9748-a77d-4181-8202-6edb393a4882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07de9748-a77d-4181-8202-6edb393a4882-istio-podinfo\") pod \"07de9748-a77d-4181-8202-6edb393a4882\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " Oct 30 05:10:13.409068 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.408915 4793 scope.go:122] "RemoveContainer" containerID="815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682" Oct 30 05:10:13.409244 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.409046 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-kube-api-access-glcs2\" (UniqueName: \"kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-kube-api-access-glcs2\") pod \"07de9748-a77d-4181-8202-6edb393a4882\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " Oct 30 05:10:13.409333 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.409242 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-data\") pod \"07de9748-a77d-4181-8202-6edb393a4882\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " Oct 30 05:10:13.409414 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.409385 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07de9748-a77d-4181-8202-6edb393a4882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07de9748-a77d-4181-8202-6edb393a4882-istiod-ca-cert\") pod \"07de9748-a77d-4181-8202-6edb393a4882\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " Oct 30 05:10:13.409529 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.409446 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-certs\") pod \"07de9748-a77d-4181-8202-6edb393a4882\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " Oct 30 05:10:13.409529 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.409504 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-credential-socket\") pod \"07de9748-a77d-4181-8202-6edb393a4882\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " Oct 30 05:10:13.409636 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.409558 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-socket\") pod \"07de9748-a77d-4181-8202-6edb393a4882\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " Oct 30 05:10:13.409685 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.409636 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-data" pod "07de9748-a77d-4181-8202-6edb393a4882" (UID: "07de9748-a77d-4181-8202-6edb393a4882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:13.409685 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.409656 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-istio-token\" (UniqueName: \"kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-istio-token\") pod \"07de9748-a77d-4181-8202-6edb393a4882\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " Oct 30 05:10:13.409806 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.409788 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-envoy\") pod \"07de9748-a77d-4181-8202-6edb393a4882\" (UID: \"07de9748-a77d-4181-8202-6edb393a4882\") " Oct 30 05:10:13.409862 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.409834 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-certs" pod "07de9748-a77d-4181-8202-6edb393a4882" (UID: "07de9748-a77d-4181-8202-6edb393a4882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:13.410163 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.410142 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07de9748-a77d-4181-8202-6edb393a4882-istiod-ca-cert" pod "07de9748-a77d-4181-8202-6edb393a4882" (UID: "07de9748-a77d-4181-8202-6edb393a4882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:13.410397 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.410378 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:13.410498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.410405 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07de9748-a77d-4181-8202-6edb393a4882-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:13.410498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.410436 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:13.410836 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:10:13.410683 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682\": container with ID starting with 815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682 not found: ID does not exist" containerID="815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682" Oct 30 05:10:13.410836 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.410723 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682"} err="failed to get container status \"815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682\": rpc error: code = NotFound desc = could not find container \"815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682\": container with ID starting with 815740e09de5656badf0cd6d01bb67d9bfa3708d9a04e24b4df0d420f37ca682 not found: ID does not exist" Oct 30 05:10:13.410836 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.410750 4793 scope.go:122] "RemoveContainer" containerID="ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba" Oct 30 05:10:13.410836 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.410826 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-credential-socket" pod "07de9748-a77d-4181-8202-6edb393a4882" (UID: "07de9748-a77d-4181-8202-6edb393a4882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:13.411129 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.410880 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-socket" pod "07de9748-a77d-4181-8202-6edb393a4882" (UID: "07de9748-a77d-4181-8202-6edb393a4882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:13.411188 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:10:13.411125 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba\": container with ID starting with ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba not found: ID does not exist" containerID="ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba" Oct 30 05:10:13.411188 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.411158 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba"} err="failed to get container status \"ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba\": rpc error: code = NotFound desc = could not find container \"ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba\": container with ID starting with ff245110f53a7f465de401e86ce9abc322e5f8efb207b41dfb3ce4d03f78b5ba not found: ID does not exist" Oct 30 05:10:13.412803 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.412777 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-envoy" pod "07de9748-a77d-4181-8202-6edb393a4882" (UID: "07de9748-a77d-4181-8202-6edb393a4882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:13.413127 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.413108 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07de9748-a77d-4181-8202-6edb393a4882-istio-podinfo" pod "07de9748-a77d-4181-8202-6edb393a4882" (UID: "07de9748-a77d-4181-8202-6edb393a4882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:10:13.413331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.413308 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-istio-token" pod "07de9748-a77d-4181-8202-6edb393a4882" (UID: "07de9748-a77d-4181-8202-6edb393a4882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:13.414516 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.414489 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-kube-api-access-glcs2" pod "07de9748-a77d-4181-8202-6edb393a4882" (UID: "07de9748-a77d-4181-8202-6edb393a4882"). InnerVolumeSpecName "kube-api-access-glcs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:13.512029 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.511994 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:13.512029 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.512024 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:13.512433 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.512039 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:13.512433 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.512051 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07de9748-a77d-4181-8202-6edb393a4882-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:13.512433 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.512062 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07de9748-a77d-4181-8202-6edb393a4882-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:13.512433 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.512075 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glcs2\" (UniqueName: \"kubernetes.io/projected/07de9748-a77d-4181-8202-6edb393a4882-kube-api-access-glcs2\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:13.716960 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.716929 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml"] Oct 30 05:10:13.728191 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:13.728167 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6378cc-openshift-default-95b887bfc-rsrml"] Oct 30 05:10:14.146279 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:14.146245 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:14.849932 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:14.849900 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07de9748-a77d-4181-8202-6edb393a4882" path="/var/lib/kubelet/pods/07de9748-a77d-4181-8202-6edb393a4882/volumes" Oct 30 05:10:14.894784 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:14.894749 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:15.118529 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.118434 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:15.646722 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.646700 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 05:10:15.765971 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.765925 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-podinfo\") pod \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " Oct 30 05:10:15.766181 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.765989 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-kube-api-access-x9w8v\" (UniqueName: \"kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-kube-api-access-x9w8v\") pod \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " Oct 30 05:10:15.766181 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.766009 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-data\") pod \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " Oct 30 05:10:15.766181 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.766100 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-socket\") pod \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " Oct 30 05:10:15.766181 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.766175 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istiod-ca-cert\") pod \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " Oct 30 05:10:15.766409 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.766211 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-token\" (UniqueName: \"kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-token\") pod \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " Oct 30 05:10:15.766409 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.766253 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-envoy\") pod \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " Oct 30 05:10:15.766409 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.766353 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-certs\") pod \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " Oct 30 05:10:15.766590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.766409 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-data" pod "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" (UID: "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.766590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.766466 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-socket" pod "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" (UID: "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.766590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.766472 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-credential-socket\") pod \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\" (UID: \"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e\") " Oct 30 05:10:15.766797 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.766683 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-credential-socket" pod "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" (UID: "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.766797 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.766787 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istiod-ca-cert" pod "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" (UID: "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:15.767005 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.766972 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-certs" pod "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" (UID: "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.767306 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.767285 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.767395 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.767314 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.767395 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.767330 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.767395 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.767344 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.767395 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.767358 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.768860 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.768815 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-envoy" pod "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" (UID: "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:10:15.768860 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.768834 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-kube-api-access-x9w8v" pod "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" (UID: "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e"). InnerVolumeSpecName "kube-api-access-x9w8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:15.769197 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.769171 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-podinfo" pod "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" (UID: "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:10:15.769313 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.769293 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-token" pod "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" (UID: "4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:15.869011 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.868985 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.869011 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.869010 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.869374 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.869019 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:15.869374 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:15.869029 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9w8v\" (UniqueName: \"kubernetes.io/projected/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e-kube-api-access-x9w8v\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:16.045721 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.045633 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:16.352213 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.352121 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:16.430830 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.430798 4793 generic.go:356] "Generic (PLEG): container finished" podID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerID="5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b" exitCode=0 Oct 30 05:10:16.431018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.430872 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" Oct 30 05:10:16.431018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.430890 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" event={"ID":"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e","Type":"ContainerDied","Data":"5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b"} Oct 30 05:10:16.431018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.430946 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl" event={"ID":"4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e","Type":"ContainerDied","Data":"86be27ef33979621c1e4490130757fc2d931b9e7daabda5f482d53d28b85e4bb"} Oct 30 05:10:16.431018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.430973 4793 scope.go:122] "RemoveContainer" containerID="5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b" Oct 30 05:10:16.443179 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.442636 4793 scope.go:122] "RemoveContainer" containerID="026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56" Oct 30 05:10:16.453231 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.453214 4793 scope.go:122] "RemoveContainer" containerID="5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b" Oct 30 05:10:16.453549 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:10:16.453524 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b\": container with ID starting with 5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b not found: ID does not exist" containerID="5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b" Oct 30 05:10:16.453655 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.453557 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b"} err="failed to get container status \"5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b\": rpc error: code = NotFound desc = could not find container \"5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b\": container with ID starting with 5b1eb21c5f4ffeb7a11900bb32f712603f117ffa345789b39b484f3bd9a80b4b not found: ID does not exist" Oct 30 05:10:16.453655 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.453583 4793 scope.go:122] "RemoveContainer" containerID="026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56" Oct 30 05:10:16.453899 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:10:16.453873 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56\": container with ID starting with 026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56 not found: ID does not exist" containerID="026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56" Oct 30 05:10:16.454006 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.453908 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56"} err="failed to get container status \"026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56\": rpc error: code = NotFound desc = could not find container \"026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56\": container with ID starting with 026dbf5bb05b95dd04a4e8357caf29bd77ced1a5874b6ced94fb948a532aed56 not found: ID does not exist" Oct 30 05:10:16.496809 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.496778 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl"] Oct 30 05:10:16.511063 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.511038 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-483bea-openshift-default-58749c6765-brcvl"] Oct 30 05:10:16.852753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:16.852716 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" path="/var/lib/kubelet/pods/4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e/volumes" Oct 30 05:10:17.498441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:17.497481 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz"] Oct 30 05:10:17.525491 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:17.525455 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2"] Oct 30 05:10:17.739868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:17.737472 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:17.940829 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:17.939825 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:18.178438 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:18.178184 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:19.525471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:19.525442 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:19.813151 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:19.813048 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:21.603765 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:21.603730 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:22.498090 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:22.498047 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" podUID="8ffbf8ef-592b-4543-9e5f-fe5b526e0281" containerName="payload-processing" containerID="cri-o://7babdc1e8e780bc3c6ae97bb8cc0f244a1dec793dd799d4e6aef57ebf69d3c81" gracePeriod=25 Oct 30 05:10:22.526524 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:22.526480 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" podUID="3a1419a8-5951-4bd3-945f-7f68f18cc7e6" containerName="payload-pre-processing" containerID="cri-o://1d1ca75585a0bc534ab8c3c33730bb4a9f4e4d6420decab7019d2352eacad6e6" gracePeriod=25 Oct 30 05:10:22.997086 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:22.997053 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2"] Oct 30 05:10:22.997529 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:22.997255 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" podUID="a585f87f-a29f-4928-a8b2-ca69834ac97d" containerName="maas-api" containerID="cri-o://87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf" gracePeriod=30 Oct 30 05:10:23.065640 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.065596 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:23.317455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.315276 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:23.343437 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.340056 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 05:10:23.529137 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.528940 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a585f87f-a29f-4928-a8b2-ca69834ac97d-kube-api-access-jb77h\" (UniqueName: \"kubernetes.io/projected/a585f87f-a29f-4928-a8b2-ca69834ac97d-kube-api-access-jb77h\") pod \"a585f87f-a29f-4928-a8b2-ca69834ac97d\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " Oct 30 05:10:23.529485 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.529383 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-metrics-certs\") pod \"a585f87f-a29f-4928-a8b2-ca69834ac97d\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " Oct 30 05:10:23.530011 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.529927 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a585f87f-a29f-4928-a8b2-ca69834ac97d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a585f87f-a29f-4928-a8b2-ca69834ac97d-openshift-service-ca\") pod \"a585f87f-a29f-4928-a8b2-ca69834ac97d\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " Oct 30 05:10:23.530446 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.530324 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-maas-api-tls\") pod \"a585f87f-a29f-4928-a8b2-ca69834ac97d\" (UID: \"a585f87f-a29f-4928-a8b2-ca69834ac97d\") " Oct 30 05:10:23.534148 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.534120 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a585f87f-a29f-4928-a8b2-ca69834ac97d-openshift-service-ca" pod "a585f87f-a29f-4928-a8b2-ca69834ac97d" (UID: "a585f87f-a29f-4928-a8b2-ca69834ac97d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:23.540408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.536898 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-metrics-certs" pod "a585f87f-a29f-4928-a8b2-ca69834ac97d" (UID: "a585f87f-a29f-4928-a8b2-ca69834ac97d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:10:23.544048 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.544020 4793 generic.go:356] "Generic (PLEG): container finished" podID="a585f87f-a29f-4928-a8b2-ca69834ac97d" containerID="87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf" exitCode=0 Oct 30 05:10:23.544162 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.544118 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" event={"ID":"a585f87f-a29f-4928-a8b2-ca69834ac97d","Type":"ContainerDied","Data":"87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf"} Oct 30 05:10:23.544236 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.544173 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" event={"ID":"a585f87f-a29f-4928-a8b2-ca69834ac97d","Type":"ContainerDied","Data":"6c48fd2e4eed15ae8d9a8af731c74eb017e9cf8119702a33eb6d1a1a3c75de07"} Oct 30 05:10:23.544236 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.544203 4793 scope.go:122] "RemoveContainer" containerID="87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf" Oct 30 05:10:23.544382 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.544367 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2" Oct 30 05:10:23.552345 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.552323 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-maas-api-tls" pod "a585f87f-a29f-4928-a8b2-ca69834ac97d" (UID: "a585f87f-a29f-4928-a8b2-ca69834ac97d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:10:23.574963 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.574930 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a585f87f-a29f-4928-a8b2-ca69834ac97d-kube-api-access-jb77h" pod "a585f87f-a29f-4928-a8b2-ca69834ac97d" (UID: "a585f87f-a29f-4928-a8b2-ca69834ac97d"). InnerVolumeSpecName "kube-api-access-jb77h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:23.580620 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.580288 4793 scope.go:122] "RemoveContainer" containerID="87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf" Oct 30 05:10:23.580967 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:10:23.580849 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf\": container with ID starting with 87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf not found: ID does not exist" containerID="87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf" Oct 30 05:10:23.580967 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.580885 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf"} err="failed to get container status \"87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf\": rpc error: code = NotFound desc = could not find container \"87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf\": container with ID starting with 87af9ed71ffad464b3a12d7fb467643d4702c747e028e36055f7b69e2e65feaf not found: ID does not exist" Oct 30 05:10:23.633116 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.633085 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:23.633288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.633122 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb77h\" (UniqueName: \"kubernetes.io/projected/a585f87f-a29f-4928-a8b2-ca69834ac97d-kube-api-access-jb77h\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:23.633288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.633135 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a585f87f-a29f-4928-a8b2-ca69834ac97d-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:23.633288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.633146 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a585f87f-a29f-4928-a8b2-ca69834ac97d-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:23.902524 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.902491 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2"] Oct 30 05:10:23.903935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.903903 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:23.909398 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:23.909378 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-fcd6a6-54b7767f78-xspw2"] Oct 30 05:10:24.847562 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:24.847523 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a585f87f-a29f-4928-a8b2-ca69834ac97d" path="/var/lib/kubelet/pods/a585f87f-a29f-4928-a8b2-ca69834ac97d/volumes" Oct 30 05:10:25.569959 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:25.569914 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:25.954715 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:25.954670 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:26.174642 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:26.174605 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:26.722482 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:26.722393 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:27.036503 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:27.036254 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:27.465256 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:27.465217 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:27.698575 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:27.698531 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:27.895483 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:27.895364 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:29.194822 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:29.194764 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:30.972030 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:30.971988 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:32.410931 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:32.410886 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" podUID="3a1419a8-5951-4bd3-945f-7f68f18cc7e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:10:32.410931 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:32.410920 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" podUID="8ffbf8ef-592b-4543-9e5f-fe5b526e0281" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:10:32.836439 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:32.836385 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:33.416118 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:33.416077 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:33.616849 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:33.616815 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:34.371023 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:34.370979 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:34.595934 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:34.595903 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:34.615771 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:34.615743 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f"] Oct 30 05:10:34.654798 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:34.654731 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t"] Oct 30 05:10:34.861058 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:34.861023 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:35.070751 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:35.070706 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:35.289440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:35.288456 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:36.140575 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:36.140540 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:36.797594 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:36.797558 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:38.728098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:38.728050 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:39.616235 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:39.616151 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" podUID="3e83024f-5869-4dcd-868a-8c2afe7ff9af" containerName="payload-processing" containerID="cri-o://a547086e3872fcb553f015b06936bf6912019308b9a9d95e89975b11bbaeb44f" gracePeriod=25 Oct 30 05:10:39.655686 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:39.655630 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" podUID="402fa222-73dc-4da9-960e-56661d43ddbb" containerName="payload-pre-processing" containerID="cri-o://3be3b0e05877f32a92f93a57e0ae710d92c2de1a6b7f3082b15c2e5b37bdfe7a" gracePeriod=25 Oct 30 05:10:40.410930 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:40.410894 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:41.076135 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:41.076103 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:41.220598 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:10:41.220564 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:10:41.340100 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:41.340011 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:41.546627 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:41.546589 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:41.653008 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:41.652928 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" podUID="402fa222-73dc-4da9-960e-56661d43ddbb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:10:42.101132 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.101093 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:42.300409 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.300370 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:42.410695 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.410610 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" podUID="3a1419a8-5951-4bd3-945f-7f68f18cc7e6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:10:42.410695 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.410649 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" podUID="8ffbf8ef-592b-4543-9e5f-fe5b526e0281" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:10:42.525323 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.525266 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:42.672893 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.672795 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" podUID="3e83024f-5869-4dcd-868a-8c2afe7ff9af" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:10:42.734573 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.731059 4793 generic.go:356] "Generic (PLEG): container finished" podID="3a1419a8-5951-4bd3-945f-7f68f18cc7e6" containerID="1d1ca75585a0bc534ab8c3c33730bb4a9f4e4d6420decab7019d2352eacad6e6" exitCode=0 Oct 30 05:10:42.734573 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.731130 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" event={"ID":"3a1419a8-5951-4bd3-945f-7f68f18cc7e6","Type":"ContainerDied","Data":"1d1ca75585a0bc534ab8c3c33730bb4a9f4e4d6420decab7019d2352eacad6e6"} Oct 30 05:10:42.734573 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.733641 4793 generic.go:356] "Generic (PLEG): container finished" podID="8ffbf8ef-592b-4543-9e5f-fe5b526e0281" containerID="7babdc1e8e780bc3c6ae97bb8cc0f244a1dec793dd799d4e6aef57ebf69d3c81" exitCode=0 Oct 30 05:10:42.734573 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.734311 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" event={"ID":"8ffbf8ef-592b-4543-9e5f-fe5b526e0281","Type":"ContainerDied","Data":"7babdc1e8e780bc3c6ae97bb8cc0f244a1dec793dd799d4e6aef57ebf69d3c81"} Oct 30 05:10:42.838931 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.838902 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" Oct 30 05:10:42.868648 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.868627 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" Oct 30 05:10:42.986148 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.986119 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-plugins-config-volume\") pod \"8ffbf8ef-592b-4543-9e5f-fe5b526e0281\" (UID: \"8ffbf8ef-592b-4543-9e5f-fe5b526e0281\") " Oct 30 05:10:42.986356 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.986202 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-kube-api-access-rpdqz\" (UniqueName: \"kubernetes.io/projected/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-kube-api-access-rpdqz\") pod \"3a1419a8-5951-4bd3-945f-7f68f18cc7e6\" (UID: \"3a1419a8-5951-4bd3-945f-7f68f18cc7e6\") " Oct 30 05:10:42.986356 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.986252 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-kube-api-access-wtt8h\" (UniqueName: \"kubernetes.io/projected/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-kube-api-access-wtt8h\") pod \"8ffbf8ef-592b-4543-9e5f-fe5b526e0281\" (UID: \"8ffbf8ef-592b-4543-9e5f-fe5b526e0281\") " Oct 30 05:10:42.986356 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.986345 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-plugins-config-volume\") pod \"3a1419a8-5951-4bd3-945f-7f68f18cc7e6\" (UID: \"3a1419a8-5951-4bd3-945f-7f68f18cc7e6\") " Oct 30 05:10:42.986602 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.986574 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-plugins-config-volume" pod "8ffbf8ef-592b-4543-9e5f-fe5b526e0281" (UID: "8ffbf8ef-592b-4543-9e5f-fe5b526e0281"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:42.986934 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.986909 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-plugins-config-volume" pod "3a1419a8-5951-4bd3-945f-7f68f18cc7e6" (UID: "3a1419a8-5951-4bd3-945f-7f68f18cc7e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:42.987047 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.987009 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:42.987047 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.987031 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:42.989056 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.989028 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-kube-api-access-wtt8h" pod "8ffbf8ef-592b-4543-9e5f-fe5b526e0281" (UID: "8ffbf8ef-592b-4543-9e5f-fe5b526e0281"). InnerVolumeSpecName "kube-api-access-wtt8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:42.989490 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.989468 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-kube-api-access-rpdqz" pod "3a1419a8-5951-4bd3-945f-7f68f18cc7e6" (UID: "3a1419a8-5951-4bd3-945f-7f68f18cc7e6"). InnerVolumeSpecName "kube-api-access-rpdqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:42.992264 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:42.992237 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:43.088304 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.088277 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpdqz\" (UniqueName: \"kubernetes.io/projected/3a1419a8-5951-4bd3-945f-7f68f18cc7e6-kube-api-access-rpdqz\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:43.088304 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.088303 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtt8h\" (UniqueName: \"kubernetes.io/projected/8ffbf8ef-592b-4543-9e5f-fe5b526e0281-kube-api-access-wtt8h\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:43.743250 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.743220 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" Oct 30 05:10:43.743766 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.743263 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz" event={"ID":"8ffbf8ef-592b-4543-9e5f-fe5b526e0281","Type":"ContainerDied","Data":"90e5fe1dbdb51e4b5bdc27ea71edc283ff8581f47edb0a3c077d47a08d698669"} Oct 30 05:10:43.743766 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.743333 4793 scope.go:122] "RemoveContainer" containerID="7babdc1e8e780bc3c6ae97bb8cc0f244a1dec793dd799d4e6aef57ebf69d3c81" Oct 30 05:10:43.744814 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.744798 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" Oct 30 05:10:43.744906 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.744834 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2" event={"ID":"3a1419a8-5951-4bd3-945f-7f68f18cc7e6","Type":"ContainerDied","Data":"acad8e947c16bfa3288b558e04be47ce798d37faef19e56ce93398cfc6e4a2e2"} Oct 30 05:10:43.755210 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.755190 4793 scope.go:122] "RemoveContainer" containerID="1d1ca75585a0bc534ab8c3c33730bb4a9f4e4d6420decab7019d2352eacad6e6" Oct 30 05:10:43.809448 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.809377 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz"] Oct 30 05:10:43.820482 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.820460 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-fcd6a6-57b96fb768-gklpz"] Oct 30 05:10:43.845529 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.845504 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2"] Oct 30 05:10:43.851735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.851714 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-fcd6a6-759c684447-6hln2"] Oct 30 05:10:43.973186 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:43.973139 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:44.850343 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:44.850305 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1419a8-5951-4bd3-945f-7f68f18cc7e6" path="/var/lib/kubelet/pods/3a1419a8-5951-4bd3-945f-7f68f18cc7e6/volumes" Oct 30 05:10:44.851013 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:44.850989 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ffbf8ef-592b-4543-9e5f-fe5b526e0281" path="/var/lib/kubelet/pods/8ffbf8ef-592b-4543-9e5f-fe5b526e0281/volumes" Oct 30 05:10:45.691098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:45.691034 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:47.513220 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:47.513186 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:47.847349 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:47.847276 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx"] Oct 30 05:10:47.863705 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:47.863670 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq"] Oct 30 05:10:48.060174 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:48.060132 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:48.683646 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:48.683600 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:48.965003 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:48.964849 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:49.184222 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.184186 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p"] Oct 30 05:10:49.184466 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.184435 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" podUID="757c4374-aa95-499b-b5e4-32c40a00ec1f" containerName="maas-api" containerID="cri-o://0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2" gracePeriod=30 Oct 30 05:10:49.332848 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.332813 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:49.597491 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.591160 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 05:10:49.683111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.681809 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757c4374-aa95-499b-b5e4-32c40a00ec1f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757c4374-aa95-499b-b5e4-32c40a00ec1f-openshift-service-ca\") pod \"757c4374-aa95-499b-b5e4-32c40a00ec1f\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " Oct 30 05:10:49.683111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.681867 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757c4374-aa95-499b-b5e4-32c40a00ec1f-kube-api-access-vwwk9\" (UniqueName: \"kubernetes.io/projected/757c4374-aa95-499b-b5e4-32c40a00ec1f-kube-api-access-vwwk9\") pod \"757c4374-aa95-499b-b5e4-32c40a00ec1f\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " Oct 30 05:10:49.683111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.681954 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-maas-api-tls\") pod \"757c4374-aa95-499b-b5e4-32c40a00ec1f\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " Oct 30 05:10:49.683111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.682086 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-metrics-certs\") pod \"757c4374-aa95-499b-b5e4-32c40a00ec1f\" (UID: \"757c4374-aa95-499b-b5e4-32c40a00ec1f\") " Oct 30 05:10:49.684507 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.684193 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757c4374-aa95-499b-b5e4-32c40a00ec1f-openshift-service-ca" pod "757c4374-aa95-499b-b5e4-32c40a00ec1f" (UID: "757c4374-aa95-499b-b5e4-32c40a00ec1f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:10:49.750805 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.747710 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757c4374-aa95-499b-b5e4-32c40a00ec1f-kube-api-access-vwwk9" pod "757c4374-aa95-499b-b5e4-32c40a00ec1f" (UID: "757c4374-aa95-499b-b5e4-32c40a00ec1f"). InnerVolumeSpecName "kube-api-access-vwwk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:10:49.750805 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.749113 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-maas-api-tls" pod "757c4374-aa95-499b-b5e4-32c40a00ec1f" (UID: "757c4374-aa95-499b-b5e4-32c40a00ec1f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:10:49.750805 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.749475 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-metrics-certs" pod "757c4374-aa95-499b-b5e4-32c40a00ec1f" (UID: "757c4374-aa95-499b-b5e4-32c40a00ec1f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:10:49.784469 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.783390 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:49.784469 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.783455 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/757c4374-aa95-499b-b5e4-32c40a00ec1f-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:49.784469 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.783473 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/757c4374-aa95-499b-b5e4-32c40a00ec1f-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:49.784469 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.783488 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwwk9\" (UniqueName: \"kubernetes.io/projected/757c4374-aa95-499b-b5e4-32c40a00ec1f-kube-api-access-vwwk9\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:10:49.811817 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.811778 4793 generic.go:356] "Generic (PLEG): container finished" podID="757c4374-aa95-499b-b5e4-32c40a00ec1f" containerID="0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2" exitCode=0 Oct 30 05:10:49.811817 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.811823 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" Oct 30 05:10:49.812053 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.811884 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" event={"ID":"757c4374-aa95-499b-b5e4-32c40a00ec1f","Type":"ContainerDied","Data":"0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2"} Oct 30 05:10:49.812053 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.811975 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p" event={"ID":"757c4374-aa95-499b-b5e4-32c40a00ec1f","Type":"ContainerDied","Data":"931af0126317af8349f5a43626967a022984be8a63d3aa4484e71e78a2a0f422"} Oct 30 05:10:49.812053 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.812003 4793 scope.go:122] "RemoveContainer" containerID="0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2" Oct 30 05:10:49.826230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.826208 4793 scope.go:122] "RemoveContainer" containerID="0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2" Oct 30 05:10:49.826549 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:10:49.826524 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2\": container with ID starting with 0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2 not found: ID does not exist" containerID="0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2" Oct 30 05:10:49.826649 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.826561 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2"} err="failed to get container status \"0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2\": rpc error: code = NotFound desc = could not find container \"0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2\": container with ID starting with 0db1b4643c1164ecd7757eb9d1be48829fcc3d804b32516c187b03bd339bb8a2 not found: ID does not exist" Oct 30 05:10:49.866651 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.866550 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p"] Oct 30 05:10:49.871234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:49.871202 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-503804-85cd6fcbbc-76k9p"] Oct 30 05:10:50.095082 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:50.095046 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:50.848244 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:50.848213 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757c4374-aa95-499b-b5e4-32c40a00ec1f" path="/var/lib/kubelet/pods/757c4374-aa95-499b-b5e4-32c40a00ec1f/volumes" Oct 30 05:10:51.652762 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:51.652722 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" podUID="402fa222-73dc-4da9-960e-56661d43ddbb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:10:51.832685 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:51.832648 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:52.661722 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:52.661686 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" podUID="3e83024f-5869-4dcd-868a-8c2afe7ff9af" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:10:53.500233 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:53.500193 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:55.370006 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:55.369954 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:56.443828 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:56.443792 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:56.671780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:56.671746 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:56.905372 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:56.905286 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:57.454355 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:57.454314 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:58.573077 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:58.573022 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:58.802933 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:58.802875 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:59.037058 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:59.037027 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:59.388112 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:59.388011 4793 scope.go:122] "RemoveContainer" containerID="db300770b387802d795c0985eaee803570a8b259fb6c7e89ca0ad8e05569bc44" Oct 30 05:10:59.394505 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:59.394474 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:59.609963 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:59.609922 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:10:59.805856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:59.803338 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl"] Oct 30 05:10:59.805856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:59.804132 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" containerID="cri-o://d8fd3e6a7d41662c51e92677cd0aa092c8210a4ede48f18077bf8dee6fde9bee" gracePeriod=30 Oct 30 05:10:59.898541 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:59.894853 4793 generic.go:356] "Generic (PLEG): container finished" podID="402fa222-73dc-4da9-960e-56661d43ddbb" containerID="3be3b0e05877f32a92f93a57e0ae710d92c2de1a6b7f3082b15c2e5b37bdfe7a" exitCode=0 Oct 30 05:10:59.898541 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:59.895014 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" event={"ID":"402fa222-73dc-4da9-960e-56661d43ddbb","Type":"ContainerDied","Data":"3be3b0e05877f32a92f93a57e0ae710d92c2de1a6b7f3082b15c2e5b37bdfe7a"} Oct 30 05:10:59.898541 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:59.897444 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" event={"ID":"3e83024f-5869-4dcd-868a-8c2afe7ff9af","Type":"ContainerDied","Data":"a547086e3872fcb553f015b06936bf6912019308b9a9d95e89975b11bbaeb44f"} Oct 30 05:10:59.898541 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:10:59.897516 4793 generic.go:356] "Generic (PLEG): container finished" podID="3e83024f-5869-4dcd-868a-8c2afe7ff9af" containerID="a547086e3872fcb553f015b06936bf6912019308b9a9d95e89975b11bbaeb44f" exitCode=0 Oct 30 05:11:00.018903 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.018814 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:00.090450 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.090406 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" Oct 30 05:11:00.134901 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.134878 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" Oct 30 05:11:00.169332 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.166517 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402fa222-73dc-4da9-960e-56661d43ddbb-kube-api-access-jm5pb\" (UniqueName: \"kubernetes.io/projected/402fa222-73dc-4da9-960e-56661d43ddbb-kube-api-access-jm5pb\") pod \"402fa222-73dc-4da9-960e-56661d43ddbb\" (UID: \"402fa222-73dc-4da9-960e-56661d43ddbb\") " Oct 30 05:11:00.169332 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.166806 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402fa222-73dc-4da9-960e-56661d43ddbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402fa222-73dc-4da9-960e-56661d43ddbb-plugins-config-volume\") pod \"402fa222-73dc-4da9-960e-56661d43ddbb\" (UID: \"402fa222-73dc-4da9-960e-56661d43ddbb\") " Oct 30 05:11:00.169332 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.167679 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402fa222-73dc-4da9-960e-56661d43ddbb-plugins-config-volume" pod "402fa222-73dc-4da9-960e-56661d43ddbb" (UID: "402fa222-73dc-4da9-960e-56661d43ddbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:00.170978 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.170952 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402fa222-73dc-4da9-960e-56661d43ddbb-kube-api-access-jm5pb" pod "402fa222-73dc-4da9-960e-56661d43ddbb" (UID: "402fa222-73dc-4da9-960e-56661d43ddbb"). InnerVolumeSpecName "kube-api-access-jm5pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:00.268337 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.268307 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e83024f-5869-4dcd-868a-8c2afe7ff9af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e83024f-5869-4dcd-868a-8c2afe7ff9af-plugins-config-volume\") pod \"3e83024f-5869-4dcd-868a-8c2afe7ff9af\" (UID: \"3e83024f-5869-4dcd-868a-8c2afe7ff9af\") " Oct 30 05:11:00.268555 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.268487 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e83024f-5869-4dcd-868a-8c2afe7ff9af-kube-api-access-fd6b4\" (UniqueName: \"kubernetes.io/projected/3e83024f-5869-4dcd-868a-8c2afe7ff9af-kube-api-access-fd6b4\") pod \"3e83024f-5869-4dcd-868a-8c2afe7ff9af\" (UID: \"3e83024f-5869-4dcd-868a-8c2afe7ff9af\") " Oct 30 05:11:00.268758 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.268733 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e83024f-5869-4dcd-868a-8c2afe7ff9af-plugins-config-volume" pod "3e83024f-5869-4dcd-868a-8c2afe7ff9af" (UID: "3e83024f-5869-4dcd-868a-8c2afe7ff9af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:00.269022 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.268972 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e83024f-5869-4dcd-868a-8c2afe7ff9af-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:00.269022 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.268998 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402fa222-73dc-4da9-960e-56661d43ddbb-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:00.269022 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.269011 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm5pb\" (UniqueName: \"kubernetes.io/projected/402fa222-73dc-4da9-960e-56661d43ddbb-kube-api-access-jm5pb\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:00.270612 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.270588 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e83024f-5869-4dcd-868a-8c2afe7ff9af-kube-api-access-fd6b4" pod "3e83024f-5869-4dcd-868a-8c2afe7ff9af" (UID: "3e83024f-5869-4dcd-868a-8c2afe7ff9af"). InnerVolumeSpecName "kube-api-access-fd6b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:00.370210 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.370182 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd6b4\" (UniqueName: \"kubernetes.io/projected/3e83024f-5869-4dcd-868a-8c2afe7ff9af-kube-api-access-fd6b4\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:00.906882 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.906805 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" Oct 30 05:11:00.906882 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.906829 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t" event={"ID":"402fa222-73dc-4da9-960e-56661d43ddbb","Type":"ContainerDied","Data":"709ddece5744201693cd22ba543d2158c7e998f359ea09ed92ddcf47d6a0784e"} Oct 30 05:11:00.907364 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.906899 4793 scope.go:122] "RemoveContainer" containerID="3be3b0e05877f32a92f93a57e0ae710d92c2de1a6b7f3082b15c2e5b37bdfe7a" Oct 30 05:11:00.908553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.908532 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" Oct 30 05:11:00.908692 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.908550 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f" event={"ID":"3e83024f-5869-4dcd-868a-8c2afe7ff9af","Type":"ContainerDied","Data":"0133a24b163fc892af2ec07fc7c7c9de386e1a448bbcf82b922a1b213167381f"} Oct 30 05:11:00.917213 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.917194 4793 scope.go:122] "RemoveContainer" containerID="a547086e3872fcb553f015b06936bf6912019308b9a9d95e89975b11bbaeb44f" Oct 30 05:11:00.968904 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.968875 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f"] Oct 30 05:11:00.980368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:00.980341 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-66217b-75f799bf98-6zt8f"] Oct 30 05:11:01.007831 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:01.007804 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t"] Oct 30 05:11:01.011095 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:01.011069 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-66217b-dbbcd7959-7xv5t"] Oct 30 05:11:01.286882 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:01.286847 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:01.517878 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:01.517838 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:02.092281 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:02.092237 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:02.317039 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:02.317005 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:02.561444 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:02.560144 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:02.847146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:02.847055 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e83024f-5869-4dcd-868a-8c2afe7ff9af" path="/var/lib/kubelet/pods/3e83024f-5869-4dcd-868a-8c2afe7ff9af/volumes" Oct 30 05:11:02.847666 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:02.847635 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402fa222-73dc-4da9-960e-56661d43ddbb" path="/var/lib/kubelet/pods/402fa222-73dc-4da9-960e-56661d43ddbb/volumes" Oct 30 05:11:02.869582 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:02.869556 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" podUID="1715329f-b8ff-48a2-8c08-32e93a0b14da" containerName="main" containerID="cri-o://607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6" gracePeriod=45 Oct 30 05:11:02.884773 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:02.884743 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" podUID="e70bca52-d177-4b80-b29b-6e0d64549ef0" containerName="main" containerID="cri-o://0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f" gracePeriod=45 Oct 30 05:11:02.988553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:02.988524 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:03.160457 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.157828 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 05:11:03.208840 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.208812 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 05:11:03.320885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.320854 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-model-cache\") pod \"e70bca52-d177-4b80-b29b-6e0d64549ef0\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.320905 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e70bca52-d177-4b80-b29b-6e0d64549ef0-tls-certs\" (UniqueName: \"kubernetes.io/secret/e70bca52-d177-4b80-b29b-6e0d64549ef0-tls-certs\") pod \"e70bca52-d177-4b80-b29b-6e0d64549ef0\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321011 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1715329f-b8ff-48a2-8c08-32e93a0b14da-tls-certs\" (UniqueName: \"kubernetes.io/secret/1715329f-b8ff-48a2-8c08-32e93a0b14da-tls-certs\") pod \"1715329f-b8ff-48a2-8c08-32e93a0b14da\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321064 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-home\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-home\") pod \"e70bca52-d177-4b80-b29b-6e0d64549ef0\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321092 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1715329f-b8ff-48a2-8c08-32e93a0b14da-kube-api-access-f4g9r\" (UniqueName: \"kubernetes.io/projected/1715329f-b8ff-48a2-8c08-32e93a0b14da-kube-api-access-f4g9r\") pod \"1715329f-b8ff-48a2-8c08-32e93a0b14da\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321155 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-model-cache" pod "e70bca52-d177-4b80-b29b-6e0d64549ef0" (UID: "e70bca52-d177-4b80-b29b-6e0d64549ef0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321183 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-model-cache\") pod \"1715329f-b8ff-48a2-8c08-32e93a0b14da\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321255 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-tmp-dir\") pod \"1715329f-b8ff-48a2-8c08-32e93a0b14da\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321297 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-home\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-home\") pod \"1715329f-b8ff-48a2-8c08-32e93a0b14da\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321376 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-home" pod "e70bca52-d177-4b80-b29b-6e0d64549ef0" (UID: "e70bca52-d177-4b80-b29b-6e0d64549ef0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321392 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-model-cache" pod "1715329f-b8ff-48a2-8c08-32e93a0b14da" (UID: "1715329f-b8ff-48a2-8c08-32e93a0b14da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321396 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-dshm\") pod \"1715329f-b8ff-48a2-8c08-32e93a0b14da\" (UID: \"1715329f-b8ff-48a2-8c08-32e93a0b14da\") " Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321472 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-tmp-dir\") pod \"e70bca52-d177-4b80-b29b-6e0d64549ef0\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321515 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70bca52-d177-4b80-b29b-6e0d64549ef0-kube-api-access-cgzmn\" (UniqueName: \"kubernetes.io/projected/e70bca52-d177-4b80-b29b-6e0d64549ef0-kube-api-access-cgzmn\") pod \"e70bca52-d177-4b80-b29b-6e0d64549ef0\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321591 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-home" pod "1715329f-b8ff-48a2-8c08-32e93a0b14da" (UID: "1715329f-b8ff-48a2-8c08-32e93a0b14da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321602 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-dshm\") pod \"e70bca52-d177-4b80-b29b-6e0d64549ef0\" (UID: \"e70bca52-d177-4b80-b29b-6e0d64549ef0\") " Oct 30 05:11:03.321663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321626 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-tmp-dir" pod "1715329f-b8ff-48a2-8c08-32e93a0b14da" (UID: "1715329f-b8ff-48a2-8c08-32e93a0b14da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:03.322708 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.321711 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-tmp-dir" pod "e70bca52-d177-4b80-b29b-6e0d64549ef0" (UID: "e70bca52-d177-4b80-b29b-6e0d64549ef0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:03.322708 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.322309 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.322708 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.322331 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.322708 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.322345 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.322708 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.322360 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.322708 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.322373 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.322708 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.322385 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.323976 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.323901 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1715329f-b8ff-48a2-8c08-32e93a0b14da-tls-certs" pod "1715329f-b8ff-48a2-8c08-32e93a0b14da" (UID: "1715329f-b8ff-48a2-8c08-32e93a0b14da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:11:03.323976 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.323949 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1715329f-b8ff-48a2-8c08-32e93a0b14da-kube-api-access-f4g9r" pod "1715329f-b8ff-48a2-8c08-32e93a0b14da" (UID: "1715329f-b8ff-48a2-8c08-32e93a0b14da"). InnerVolumeSpecName "kube-api-access-f4g9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:03.323976 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.323950 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-dshm" pod "1715329f-b8ff-48a2-8c08-32e93a0b14da" (UID: "1715329f-b8ff-48a2-8c08-32e93a0b14da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:03.324156 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.323983 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70bca52-d177-4b80-b29b-6e0d64549ef0-tls-certs" pod "e70bca52-d177-4b80-b29b-6e0d64549ef0" (UID: "e70bca52-d177-4b80-b29b-6e0d64549ef0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:11:03.324300 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.324269 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70bca52-d177-4b80-b29b-6e0d64549ef0-kube-api-access-cgzmn" pod "e70bca52-d177-4b80-b29b-6e0d64549ef0" (UID: "e70bca52-d177-4b80-b29b-6e0d64549ef0"). InnerVolumeSpecName "kube-api-access-cgzmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:03.324544 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.324526 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-dshm" pod "e70bca52-d177-4b80-b29b-6e0d64549ef0" (UID: "e70bca52-d177-4b80-b29b-6e0d64549ef0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:03.424230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.424157 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70bca52-d177-4b80-b29b-6e0d64549ef0-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.424230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.424183 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1715329f-b8ff-48a2-8c08-32e93a0b14da-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.424230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.424191 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4g9r\" (UniqueName: \"kubernetes.io/projected/1715329f-b8ff-48a2-8c08-32e93a0b14da-kube-api-access-f4g9r\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.424230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.424201 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1715329f-b8ff-48a2-8c08-32e93a0b14da-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.424230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.424211 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgzmn\" (UniqueName: \"kubernetes.io/projected/e70bca52-d177-4b80-b29b-6e0d64549ef0-kube-api-access-cgzmn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.424230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.424220 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70bca52-d177-4b80-b29b-6e0d64549ef0-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:03.778971 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.778901 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:03.932749 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.932717 4793 generic.go:356] "Generic (PLEG): container finished" podID="1715329f-b8ff-48a2-8c08-32e93a0b14da" containerID="607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6" exitCode=0 Oct 30 05:11:03.932952 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.932758 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" Oct 30 05:11:03.932952 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.932805 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" event={"ID":"1715329f-b8ff-48a2-8c08-32e93a0b14da","Type":"ContainerDied","Data":"607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6"} Oct 30 05:11:03.932952 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.932842 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx" event={"ID":"1715329f-b8ff-48a2-8c08-32e93a0b14da","Type":"ContainerDied","Data":"357871f650e10d4ec7c4322c0a0de3180841ba2d8a95901365237672fa1043a1"} Oct 30 05:11:03.932952 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.932862 4793 scope.go:122] "RemoveContainer" containerID="607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6" Oct 30 05:11:03.934355 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.934333 4793 generic.go:356] "Generic (PLEG): container finished" podID="e70bca52-d177-4b80-b29b-6e0d64549ef0" containerID="0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f" exitCode=0 Oct 30 05:11:03.934644 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.934449 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" Oct 30 05:11:03.934644 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.934449 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" event={"ID":"e70bca52-d177-4b80-b29b-6e0d64549ef0","Type":"ContainerDied","Data":"0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f"} Oct 30 05:11:03.934644 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.934488 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq" event={"ID":"e70bca52-d177-4b80-b29b-6e0d64549ef0","Type":"ContainerDied","Data":"d812c1e62139555401751cf52e21899b921bb8b058db863e07ca3e886f67dc3e"} Oct 30 05:11:03.950063 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.949668 4793 scope.go:122] "RemoveContainer" containerID="607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6" Oct 30 05:11:03.950294 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:11:03.950175 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6\": container with ID starting with 607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6 not found: ID does not exist" containerID="607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6" Oct 30 05:11:03.950294 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.950223 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6"} err="failed to get container status \"607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6\": rpc error: code = NotFound desc = could not find container \"607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6\": container with ID starting with 607970d04ddf2049dfba986f1ad3c6b0563b2714842014dd3fd9cb7686d7cef6 not found: ID does not exist" Oct 30 05:11:03.950294 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.950256 4793 scope.go:122] "RemoveContainer" containerID="0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f" Oct 30 05:11:03.963205 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.963181 4793 scope.go:122] "RemoveContainer" containerID="0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f" Oct 30 05:11:03.963537 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:11:03.963515 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f\": container with ID starting with 0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f not found: ID does not exist" containerID="0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f" Oct 30 05:11:03.963617 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.963547 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f"} err="failed to get container status \"0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f\": rpc error: code = NotFound desc = could not find container \"0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f\": container with ID starting with 0505597154dd93d87d146ff6413c84bb800bffbe92b7027ace248b4d4e14b87f not found: ID does not exist" Oct 30 05:11:03.993167 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:03.993093 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq"] Oct 30 05:11:04.005743 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:04.005699 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/simulator-w2-5d74b9-kserve-cb6745f6-f5rjq"] Oct 30 05:11:04.033111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:04.033028 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx"] Oct 30 05:11:04.047116 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:04.047089 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5d74b9/distinct-w2-5d74b9-kserve-69b96f95bd-pltdx"] Oct 30 05:11:04.219377 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:04.219341 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:04.847396 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:04.847370 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1715329f-b8ff-48a2-8c08-32e93a0b14da" path="/var/lib/kubelet/pods/1715329f-b8ff-48a2-8c08-32e93a0b14da/volumes" Oct 30 05:11:04.847856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:04.847839 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70bca52-d177-4b80-b29b-6e0d64549ef0" path="/var/lib/kubelet/pods/e70bca52-d177-4b80-b29b-6e0d64549ef0/volumes" Oct 30 05:11:04.942862 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:04.942818 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:04.944028 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:04.944003 4793 generic.go:356] "Generic (PLEG): container finished" podID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerID="d8fd3e6a7d41662c51e92677cd0aa092c8210a4ede48f18077bf8dee6fde9bee" exitCode=0 Oct 30 05:11:04.944146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:04.944079 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" event={"ID":"943d71d5-69a1-45ba-8cb1-0532e9d15b42","Type":"ContainerDied","Data":"d8fd3e6a7d41662c51e92677cd0aa092c8210a4ede48f18077bf8dee6fde9bee"} Oct 30 05:11:04.944200 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:04.944148 4793 scope.go:122] "RemoveContainer" containerID="b6d73d7858ebc0dbabcd5194bb65350ed784605831a1d16ca998fbcaf9be87c7" Oct 30 05:11:05.114223 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.114198 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:11:05.147777 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.147749 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-podinfo\") pod \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " Oct 30 05:11:05.147912 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.147805 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-envoy\") pod \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " Oct 30 05:11:05.147912 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.147877 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-data\") pod \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " Oct 30 05:11:05.148037 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.147978 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istiod-ca-cert\") pod \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " Oct 30 05:11:05.148098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148074 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-kube-api-access-bfn9z\" (UniqueName: \"kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-kube-api-access-bfn9z\") pod \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " Oct 30 05:11:05.148195 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148153 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-credential-socket\") pod \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " Oct 30 05:11:05.148195 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148151 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-data" pod "943d71d5-69a1-45ba-8cb1-0532e9d15b42" (UID: "943d71d5-69a1-45ba-8cb1-0532e9d15b42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:05.148308 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148194 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-certs\") pod \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " Oct 30 05:11:05.148308 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148242 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-socket\") pod \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " Oct 30 05:11:05.148308 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148270 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-token\" (UniqueName: \"kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-token\") pod \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\" (UID: \"943d71d5-69a1-45ba-8cb1-0532e9d15b42\") " Oct 30 05:11:05.148489 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148407 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-credential-socket" pod "943d71d5-69a1-45ba-8cb1-0532e9d15b42" (UID: "943d71d5-69a1-45ba-8cb1-0532e9d15b42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:05.148489 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148469 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istiod-ca-cert" pod "943d71d5-69a1-45ba-8cb1-0532e9d15b42" (UID: "943d71d5-69a1-45ba-8cb1-0532e9d15b42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:05.148822 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148797 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-socket" pod "943d71d5-69a1-45ba-8cb1-0532e9d15b42" (UID: "943d71d5-69a1-45ba-8cb1-0532e9d15b42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:05.148941 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148846 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-certs" pod "943d71d5-69a1-45ba-8cb1-0532e9d15b42" (UID: "943d71d5-69a1-45ba-8cb1-0532e9d15b42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:05.148941 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148860 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:05.148941 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148884 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:05.148941 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.148900 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:05.150182 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.150160 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-kube-api-access-bfn9z" pod "943d71d5-69a1-45ba-8cb1-0532e9d15b42" (UID: "943d71d5-69a1-45ba-8cb1-0532e9d15b42"). InnerVolumeSpecName "kube-api-access-bfn9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:05.150346 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.150329 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-envoy" pod "943d71d5-69a1-45ba-8cb1-0532e9d15b42" (UID: "943d71d5-69a1-45ba-8cb1-0532e9d15b42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:05.150530 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.150329 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-podinfo" pod "943d71d5-69a1-45ba-8cb1-0532e9d15b42" (UID: "943d71d5-69a1-45ba-8cb1-0532e9d15b42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:11:05.150677 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.150659 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-token" pod "943d71d5-69a1-45ba-8cb1-0532e9d15b42" (UID: "943d71d5-69a1-45ba-8cb1-0532e9d15b42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:05.250237 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.250207 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfn9z\" (UniqueName: \"kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-kube-api-access-bfn9z\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:05.250237 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.250237 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:05.250668 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.250248 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:05.250668 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.250257 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:05.250668 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.250265 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:05.250668 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.250273 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943d71d5-69a1-45ba-8cb1-0532e9d15b42-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:05.713370 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.713339 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:05.955889 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.955865 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" Oct 30 05:11:05.956088 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.955902 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl" event={"ID":"943d71d5-69a1-45ba-8cb1-0532e9d15b42","Type":"ContainerDied","Data":"46d21623613361cb9f62c8ff7313411dec93c2aecd1a389e856093ecee819beb"} Oct 30 05:11:05.956088 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:05.955968 4793 scope.go:122] "RemoveContainer" containerID="d8fd3e6a7d41662c51e92677cd0aa092c8210a4ede48f18077bf8dee6fde9bee" Oct 30 05:11:06.021268 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:06.021240 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl"] Oct 30 05:11:06.033045 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:06.033018 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b5496290-openshift-default-786dddd54f-tplpl"] Oct 30 05:11:06.847414 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:06.847382 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" path="/var/lib/kubelet/pods/943d71d5-69a1-45ba-8cb1-0532e9d15b42/volumes" Oct 30 05:11:07.707460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:07.701991 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:08.045948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:08.045704 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:09.605987 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:09.605949 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:11.220846 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:11:11.220809 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:11:11.511308 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:11.511220 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:12.796867 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:12.796818 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:13.197468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:13.196708 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:14.676872 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:14.676821 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:15.373954 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:15.373912 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:17.112893 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:17.112854 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:17.450127 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:17.450092 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:18.685714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:18.685679 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:18.902657 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:18.902621 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:19.418298 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:19.418264 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:19.656988 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:19.656954 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz"] Oct 30 05:11:19.657493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:19.657457 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" podUID="e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" containerName="istio-proxy" containerID="cri-o://1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc" gracePeriod=30 Oct 30 05:11:19.792385 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:19.792352 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:20.842275 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:20.842241 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:22.599893 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:22.599858 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:24.325386 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:24.325342 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:24.776047 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:24.776012 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:24.995787 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:24.991175 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 05:11:25.014370 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.014338 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:25.103220 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.103182 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istiod-ca-cert\") pod \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " Oct 30 05:11:25.103378 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.103280 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-kube-api-access-wgfdn\" (UniqueName: \"kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-kube-api-access-wgfdn\") pod \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " Oct 30 05:11:25.103466 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.103379 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-certs\") pod \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " Oct 30 05:11:25.103518 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.103466 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-data\") pod \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " Oct 30 05:11:25.103573 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.103557 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-token\" (UniqueName: \"kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-token\") pod \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " Oct 30 05:11:25.103608 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.103590 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-podinfo\") pod \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " Oct 30 05:11:25.103653 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.103640 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-socket\") pod \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " Oct 30 05:11:25.103744 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.103730 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-credential-socket\") pod \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " Oct 30 05:11:25.103811 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.103797 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-envoy\") pod \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\" (UID: \"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4\") " Oct 30 05:11:25.105769 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.105732 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-credential-socket" pod "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" (UID: "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:25.106259 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.105949 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-socket" pod "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" (UID: "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:25.106475 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.106437 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-certs" pod "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" (UID: "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:25.106771 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.106735 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istiod-ca-cert" pod "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" (UID: "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:25.106930 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.106910 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-data" pod "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" (UID: "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:25.107200 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.107176 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-envoy" pod "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" (UID: "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:11:25.109758 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.107712 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-podinfo" pod "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" (UID: "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:11:25.109758 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.108520 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-kube-api-access-wgfdn" pod "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" (UID: "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4"). InnerVolumeSpecName "kube-api-access-wgfdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:25.109758 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.108829 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-token" pod "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" (UID: "e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:25.113138 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.113109 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8"] Oct 30 05:11:25.142855 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.142829 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778"] Oct 30 05:11:25.209470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.207547 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:25.209470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.207581 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:25.209470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.207595 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:25.209470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.207609 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:25.209470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.207624 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:25.209470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.207640 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:25.209470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.207653 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:25.209470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.207668 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgfdn\" (UniqueName: \"kubernetes.io/projected/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-kube-api-access-wgfdn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:25.209470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.207682 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:25.210571 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.210544 4793 generic.go:356] "Generic (PLEG): container finished" podID="e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" containerID="1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc" exitCode=0 Oct 30 05:11:25.211793 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.211770 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" Oct 30 05:11:25.213746 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.213655 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" event={"ID":"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4","Type":"ContainerDied","Data":"1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc"} Oct 30 05:11:25.213746 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.213715 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz" event={"ID":"e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4","Type":"ContainerDied","Data":"458b38f38e4742adf767a1b8f0df0912e6f13276ee05b3591a96bf1c070b2ee8"} Oct 30 05:11:25.213746 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.213745 4793 scope.go:122] "RemoveContainer" containerID="1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc" Oct 30 05:11:25.232338 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.232316 4793 scope.go:122] "RemoveContainer" containerID="1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc" Oct 30 05:11:25.232719 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:11:25.232690 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc\": container with ID starting with 1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc not found: ID does not exist" containerID="1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc" Oct 30 05:11:25.232819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.232732 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc"} err="failed to get container status \"1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc\": rpc error: code = NotFound desc = could not find container \"1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc\": container with ID starting with 1ab53662c2e84b87ac924316b13ee101e9d5b96e59e5a5ea8fc90d534753eedc not found: ID does not exist" Oct 30 05:11:25.238011 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.237988 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:25.269754 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.269726 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz"] Oct 30 05:11:25.279663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.279637 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5d74b9-openshift-default-c4dbcb75-2vjfz"] Oct 30 05:11:25.473730 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.473640 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:25.731850 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:25.731773 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:26.301853 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:26.301816 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:26.717831 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:26.717785 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:26.847272 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:26.847243 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" path="/var/lib/kubelet/pods/e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4/volumes" Oct 30 05:11:27.988921 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:27.988881 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:29.828846 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:29.828807 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:30.114382 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:30.114292 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" podUID="11b7ca3f-50a5-4aa2-9b53-241419d21cda" containerName="payload-processing" containerID="cri-o://527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c" gracePeriod=25 Oct 30 05:11:30.143383 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:30.143352 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" podUID="3698383d-bb0c-41c5-84d2-1b5aa813151e" containerName="payload-pre-processing" containerID="cri-o://2e37f499449e1bbf4361819b8f340524eff1ae9d7350b9cc8a7a3820fa838f85" gracePeriod=25 Oct 30 05:11:31.528009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:31.527973 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:33.013757 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:33.013724 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:33.230786 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:33.230754 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:33.683889 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:33.683849 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:34.806773 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:34.806736 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:36.160231 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:36.160187 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" podUID="11b7ca3f-50a5-4aa2-9b53-241419d21cda" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:11:36.160713 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:36.160241 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" podUID="3698383d-bb0c-41c5-84d2-1b5aa813151e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:11:36.646866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:36.646813 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:38.438110 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:38.438078 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:39.986199 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.986164 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww"] Oct 30 05:11:39.987878 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.987849 4793 cpu_manager.go:414] "containerMap: removing container" podUID="e70bca52-d177-4b80-b29b-6e0d64549ef0" containerName="main" Oct 30 05:11:39.988196 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.988034 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70bca52-d177-4b80-b29b-6e0d64549ef0" containerName="main" Oct 30 05:11:39.988360 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.988343 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2996a1cc-21e1-4025-bb05-a52f82f82958" containerName="maas-api" Oct 30 05:11:39.988479 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.988467 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2996a1cc-21e1-4025-bb05-a52f82f82958" containerName="maas-api" Oct 30 05:11:39.988582 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.988570 4793 cpu_manager.go:414] "containerMap: removing container" podUID="8ffbf8ef-592b-4543-9e5f-fe5b526e0281" containerName="payload-processing" Oct 30 05:11:39.988662 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.988651 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ffbf8ef-592b-4543-9e5f-fe5b526e0281" containerName="payload-processing" Oct 30 05:11:39.988761 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.988751 4793 cpu_manager.go:414] "containerMap: removing container" podUID="357e05d0-d457-4cb2-8d86-4a7990e6d6eb" containerName="payload-pre-processing" Oct 30 05:11:39.988838 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.988826 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357e05d0-d457-4cb2-8d86-4a7990e6d6eb" containerName="payload-pre-processing" Oct 30 05:11:39.988926 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.988916 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a585f87f-a29f-4928-a8b2-ca69834ac97d" containerName="maas-api" Oct 30 05:11:39.989009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.988998 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a585f87f-a29f-4928-a8b2-ca69834ac97d" containerName="maas-api" Oct 30 05:11:39.989102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.989092 4793 cpu_manager.go:414] "containerMap: removing container" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerName="istio-proxy" Oct 30 05:11:39.989186 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.989173 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerName="istio-proxy" Oct 30 05:11:39.989276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.989264 4793 cpu_manager.go:414] "containerMap: removing container" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" Oct 30 05:11:39.989359 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.989348 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" Oct 30 05:11:39.989464 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.989454 4793 cpu_manager.go:414] "containerMap: removing container" podUID="757c4374-aa95-499b-b5e4-32c40a00ec1f" containerName="maas-api" Oct 30 05:11:39.989551 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.989541 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757c4374-aa95-499b-b5e4-32c40a00ec1f" containerName="maas-api" Oct 30 05:11:39.989659 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.989647 4793 cpu_manager.go:414] "containerMap: removing container" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" Oct 30 05:11:39.990124 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.990104 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" Oct 30 05:11:39.990260 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.990248 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3049a33a-cac2-4174-82d4-7e0a7b96fdb0" containerName="maas-api" Oct 30 05:11:39.990341 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.990332 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3049a33a-cac2-4174-82d4-7e0a7b96fdb0" containerName="maas-api" Oct 30 05:11:39.990435 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.990411 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3a1419a8-5951-4bd3-945f-7f68f18cc7e6" containerName="payload-pre-processing" Oct 30 05:11:39.990521 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.990510 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1419a8-5951-4bd3-945f-7f68f18cc7e6" containerName="payload-pre-processing" Oct 30 05:11:39.990606 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.990597 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3e83024f-5869-4dcd-868a-8c2afe7ff9af" containerName="payload-processing" Oct 30 05:11:39.990693 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.990680 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e83024f-5869-4dcd-868a-8c2afe7ff9af" containerName="payload-processing" Oct 30 05:11:39.990783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.990774 4793 cpu_manager.go:414] "containerMap: removing container" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerName="istio-proxy" Oct 30 05:11:39.990859 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.990849 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerName="istio-proxy" Oct 30 05:11:39.990943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.990934 4793 cpu_manager.go:414] "containerMap: removing container" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" Oct 30 05:11:39.991021 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991011 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" Oct 30 05:11:39.991104 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991094 4793 cpu_manager.go:414] "containerMap: removing container" podUID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" containerName="istio-proxy" Oct 30 05:11:39.991180 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991170 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" containerName="istio-proxy" Oct 30 05:11:39.991264 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991255 4793 cpu_manager.go:414] "containerMap: removing container" podUID="e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" containerName="istio-proxy" Oct 30 05:11:39.991346 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991336 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" containerName="istio-proxy" Oct 30 05:11:39.991458 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991432 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1715329f-b8ff-48a2-8c08-32e93a0b14da" containerName="main" Oct 30 05:11:39.991555 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991544 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1715329f-b8ff-48a2-8c08-32e93a0b14da" containerName="main" Oct 30 05:11:39.991638 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991627 4793 cpu_manager.go:414] "containerMap: removing container" podUID="402fa222-73dc-4da9-960e-56661d43ddbb" containerName="payload-pre-processing" Oct 30 05:11:39.991712 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991695 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402fa222-73dc-4da9-960e-56661d43ddbb" containerName="payload-pre-processing" Oct 30 05:11:39.991788 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991719 4793 cpu_manager.go:414] "containerMap: removing container" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" Oct 30 05:11:39.991788 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991728 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" Oct 30 05:11:39.991901 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991889 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b29e1f-12ff-4b1a-8fbc-52e926ce5ba4" containerName="istio-proxy" Oct 30 05:11:39.991966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991902 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1715329f-b8ff-48a2-8c08-32e93a0b14da" containerName="main" Oct 30 05:11:39.991966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991912 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="402fa222-73dc-4da9-960e-56661d43ddbb" containerName="payload-pre-processing" Oct 30 05:11:39.991966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991922 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" Oct 30 05:11:39.991966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991932 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70bca52-d177-4b80-b29b-6e0d64549ef0" containerName="main" Oct 30 05:11:39.991966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991942 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="65430b4d-69e5-4e30-8a47-9c1e9003759c" containerName="istio-proxy" Oct 30 05:11:39.991966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991953 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2996a1cc-21e1-4025-bb05-a52f82f82958" containerName="maas-api" Oct 30 05:11:39.991966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991966 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ffbf8ef-592b-4543-9e5f-fe5b526e0281" containerName="payload-processing" Oct 30 05:11:39.992328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991976 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="357e05d0-d457-4cb2-8d86-4a7990e6d6eb" containerName="payload-pre-processing" Oct 30 05:11:39.992328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991988 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a585f87f-a29f-4928-a8b2-ca69834ac97d" containerName="maas-api" Oct 30 05:11:39.992328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.991997 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerName="istio-proxy" Oct 30 05:11:39.992328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.992005 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" Oct 30 05:11:39.992328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.992014 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="757c4374-aa95-499b-b5e4-32c40a00ec1f" containerName="maas-api" Oct 30 05:11:39.992328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.992025 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="07de9748-a77d-4181-8202-6edb393a4882" containerName="istio-proxy" Oct 30 05:11:39.992328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.992033 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3049a33a-cac2-4174-82d4-7e0a7b96fdb0" containerName="maas-api" Oct 30 05:11:39.992328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.992044 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1419a8-5951-4bd3-945f-7f68f18cc7e6" containerName="payload-pre-processing" Oct 30 05:11:39.992328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.992054 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e83024f-5869-4dcd-868a-8c2afe7ff9af" containerName="payload-processing" Oct 30 05:11:39.992328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.992061 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="943d71d5-69a1-45ba-8cb1-0532e9d15b42" containerName="istio-proxy" Oct 30 05:11:39.992328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.992072 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f0f764-1aa5-4d4e-a4ed-a3b61e2bd6b2" containerName="istio-proxy" Oct 30 05:11:39.995865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.995843 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww"] Oct 30 05:11:39.995994 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:39.995947 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.025355 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.025323 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-495ebe\"/\"default-dockercfg-lclwq\"" Oct 30 05:11:40.025355 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.025323 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-495ebe\"/\"openshift-service-ca.crt\"" Oct 30 05:11:40.025573 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.025356 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-495ebe\"/\"kube-root-ca.crt\"" Oct 30 05:11:40.025573 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.025506 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-495ebe\"/\"rate-test-model-495ebe-kserve-self-signed-certs\"" Oct 30 05:11:40.105391 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.105357 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2tdd\" (UniqueName: \"kubernetes.io/projected/3d60ccab-2ec3-4c1c-875e-2544990823ed-kube-api-access-v2tdd\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.105581 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.105507 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-dshm\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.105581 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.105577 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d60ccab-2ec3-4c1c-875e-2544990823ed-tls-certs\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.105779 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.105717 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-home\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.105779 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.105754 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-tmp-dir\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.105897 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.105878 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-model-cache\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.207481 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.207432 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-model-cache\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.207868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.207836 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2tdd\" (UniqueName: \"kubernetes.io/projected/3d60ccab-2ec3-4c1c-875e-2544990823ed-kube-api-access-v2tdd\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.208011 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.207941 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-dshm\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.208079 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.208014 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d60ccab-2ec3-4c1c-875e-2544990823ed-tls-certs\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.208079 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.208030 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-model-cache\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.208198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.208123 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-home\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.208198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.208151 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-tmp-dir\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.208560 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.208532 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-tmp-dir\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.208720 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.208698 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-home\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.211283 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.211258 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-dshm\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.240498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.233145 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d60ccab-2ec3-4c1c-875e-2544990823ed-tls-certs\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.273705 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.271252 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:40.273705 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.273630 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2tdd\" (UniqueName: \"kubernetes.io/projected/3d60ccab-2ec3-4c1c-875e-2544990823ed-kube-api-access-v2tdd\") pod \"rate-test-model-495ebe-kserve-7d6b4c85d9-42bww\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.311143 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.311105 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:40.480727 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.480696 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:40.639646 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.639616 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww"] Oct 30 05:11:40.641388 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:11:40.641361 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d60ccab_2ec3_4c1c_875e_2544990823ed.slice/crio-aa6beeae1f858b228e1acffdbb817b3af05ca6ddb052a53e78c674f2077993b8 WatchSource:0}: Error finding container aa6beeae1f858b228e1acffdbb817b3af05ca6ddb052a53e78c674f2077993b8: Status 404 returned error can't find the container with id aa6beeae1f858b228e1acffdbb817b3af05ca6ddb052a53e78c674f2077993b8 Oct 30 05:11:40.837171 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:40.837135 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:41.221262 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:11:41.221224 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:11:41.352774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:41.352732 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" event={"ID":"3d60ccab-2ec3-4c1c-875e-2544990823ed","Type":"ContainerStarted","Data":"8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87"} Oct 30 05:11:41.352774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:41.352774 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" event={"ID":"3d60ccab-2ec3-4c1c-875e-2544990823ed","Type":"ContainerStarted","Data":"aa6beeae1f858b228e1acffdbb817b3af05ca6ddb052a53e78c674f2077993b8"} Oct 30 05:11:41.395800 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:41.395744 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" podStartSLOduration=2.395730248 podStartE2EDuration="2.395730248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:11:41.392469687 +0000 UTC m=+4424.678344464" watchObservedRunningTime="2026-10-30 05:11:41.395730248 +0000 UTC m=+4424.681605043" Oct 30 05:11:41.995694 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:41.995641 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:42.311560 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:42.311483 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:43.619984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:43.619951 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:45.233501 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:45.233454 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:45.461285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:45.461237 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:45.680775 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:45.680746 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:46.160362 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:46.160266 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" podUID="3698383d-bb0c-41c5-84d2-1b5aa813151e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:11:46.160581 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:46.160278 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" podUID="11b7ca3f-50a5-4aa2-9b53-241419d21cda" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:11:46.163608 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:46.163583 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:47.006751 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:47.006694 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:48.732532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:48.732500 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:50.311499 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.311401 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:50.330445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.329124 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:50.427559 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.427528 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" Oct 30 05:11:50.434574 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.434366 4793 generic.go:356] "Generic (PLEG): container finished" podID="3698383d-bb0c-41c5-84d2-1b5aa813151e" containerID="2e37f499449e1bbf4361819b8f340524eff1ae9d7350b9cc8a7a3820fa838f85" exitCode=0 Oct 30 05:11:50.434574 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.434494 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" event={"ID":"3698383d-bb0c-41c5-84d2-1b5aa813151e","Type":"ContainerDied","Data":"2e37f499449e1bbf4361819b8f340524eff1ae9d7350b9cc8a7a3820fa838f85"} Oct 30 05:11:50.437337 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.436695 4793 generic.go:356] "Generic (PLEG): container finished" podID="11b7ca3f-50a5-4aa2-9b53-241419d21cda" containerID="527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c" exitCode=0 Oct 30 05:11:50.438216 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.438198 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" Oct 30 05:11:50.438907 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.438766 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" event={"ID":"11b7ca3f-50a5-4aa2-9b53-241419d21cda","Type":"ContainerDied","Data":"527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c"} Oct 30 05:11:50.439478 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.439060 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8" event={"ID":"11b7ca3f-50a5-4aa2-9b53-241419d21cda","Type":"ContainerDied","Data":"c89d1aef297b2d473ea47fffd341e7de1479b607decf6ecf665349a214d9a193"} Oct 30 05:11:50.439478 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.439102 4793 scope.go:122] "RemoveContainer" containerID="527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c" Oct 30 05:11:50.447791 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.447535 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b7ca3f-50a5-4aa2-9b53-241419d21cda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b7ca3f-50a5-4aa2-9b53-241419d21cda-plugins-config-volume\") pod \"11b7ca3f-50a5-4aa2-9b53-241419d21cda\" (UID: \"11b7ca3f-50a5-4aa2-9b53-241419d21cda\") " Oct 30 05:11:50.447791 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.447771 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b7ca3f-50a5-4aa2-9b53-241419d21cda-kube-api-access-zx7t4\" (UniqueName: \"kubernetes.io/projected/11b7ca3f-50a5-4aa2-9b53-241419d21cda-kube-api-access-zx7t4\") pod \"11b7ca3f-50a5-4aa2-9b53-241419d21cda\" (UID: \"11b7ca3f-50a5-4aa2-9b53-241419d21cda\") " Oct 30 05:11:50.447962 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.447929 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b7ca3f-50a5-4aa2-9b53-241419d21cda-plugins-config-volume" pod "11b7ca3f-50a5-4aa2-9b53-241419d21cda" (UID: "11b7ca3f-50a5-4aa2-9b53-241419d21cda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:50.448077 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.448054 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b7ca3f-50a5-4aa2-9b53-241419d21cda-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:50.449943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.449917 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b7ca3f-50a5-4aa2-9b53-241419d21cda-kube-api-access-zx7t4" pod "11b7ca3f-50a5-4aa2-9b53-241419d21cda" (UID: "11b7ca3f-50a5-4aa2-9b53-241419d21cda"). InnerVolumeSpecName "kube-api-access-zx7t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:50.452621 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.452603 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:11:50.457066 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.456682 4793 scope.go:122] "RemoveContainer" containerID="527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c" Oct 30 05:11:50.457143 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:11:50.457112 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c\": container with ID starting with 527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c not found: ID does not exist" containerID="527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c" Oct 30 05:11:50.457202 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.457166 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c"} err="failed to get container status \"527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c\": rpc error: code = NotFound desc = could not find container \"527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c\": container with ID starting with 527839a78ef4a781a45247ce609737a4be72e63919fc2fb9b82632bb8ffcc51c not found: ID does not exist" Oct 30 05:11:50.508334 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.508305 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" Oct 30 05:11:50.543013 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.542969 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:50.549237 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.549204 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3698383d-bb0c-41c5-84d2-1b5aa813151e-kube-api-access-psvxq\" (UniqueName: \"kubernetes.io/projected/3698383d-bb0c-41c5-84d2-1b5aa813151e-kube-api-access-psvxq\") pod \"3698383d-bb0c-41c5-84d2-1b5aa813151e\" (UID: \"3698383d-bb0c-41c5-84d2-1b5aa813151e\") " Oct 30 05:11:50.549393 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.549322 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3698383d-bb0c-41c5-84d2-1b5aa813151e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3698383d-bb0c-41c5-84d2-1b5aa813151e-plugins-config-volume\") pod \"3698383d-bb0c-41c5-84d2-1b5aa813151e\" (UID: \"3698383d-bb0c-41c5-84d2-1b5aa813151e\") " Oct 30 05:11:50.550457 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.550396 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx7t4\" (UniqueName: \"kubernetes.io/projected/11b7ca3f-50a5-4aa2-9b53-241419d21cda-kube-api-access-zx7t4\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:50.550916 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.550888 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3698383d-bb0c-41c5-84d2-1b5aa813151e-plugins-config-volume" pod "3698383d-bb0c-41c5-84d2-1b5aa813151e" (UID: "3698383d-bb0c-41c5-84d2-1b5aa813151e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:11:50.553269 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.552581 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3698383d-bb0c-41c5-84d2-1b5aa813151e-kube-api-access-psvxq" pod "3698383d-bb0c-41c5-84d2-1b5aa813151e" (UID: "3698383d-bb0c-41c5-84d2-1b5aa813151e"). InnerVolumeSpecName "kube-api-access-psvxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:11:50.654118 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.653999 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psvxq\" (UniqueName: \"kubernetes.io/projected/3698383d-bb0c-41c5-84d2-1b5aa813151e-kube-api-access-psvxq\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:50.654118 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.654034 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3698383d-bb0c-41c5-84d2-1b5aa813151e-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:11:50.784082 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.784053 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8"] Oct 30 05:11:50.788382 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.788357 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-6796b1-7b59f6c658-vgzs8"] Oct 30 05:11:50.849676 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:50.849647 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b7ca3f-50a5-4aa2-9b53-241419d21cda" path="/var/lib/kubelet/pods/11b7ca3f-50a5-4aa2-9b53-241419d21cda/volumes" Oct 30 05:11:51.447017 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:51.446989 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" Oct 30 05:11:51.447541 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:51.446990 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778" event={"ID":"3698383d-bb0c-41c5-84d2-1b5aa813151e","Type":"ContainerDied","Data":"4eda5c96f34b8f74e1edc250321b56acae026f049781fc65710342c26c90a037"} Oct 30 05:11:51.447541 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:51.447122 4793 scope.go:122] "RemoveContainer" containerID="2e37f499449e1bbf4361819b8f340524eff1ae9d7350b9cc8a7a3820fa838f85" Oct 30 05:11:51.499433 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:51.499388 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778"] Oct 30 05:11:51.512285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:51.512258 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6796b1-768bc96979-6s778"] Oct 30 05:11:52.231514 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:52.231462 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:52.450712 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:52.450676 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:52.846474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:52.846444 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3698383d-bb0c-41c5-84d2-1b5aa813151e" path="/var/lib/kubelet/pods/3698383d-bb0c-41c5-84d2-1b5aa813151e/volumes" Oct 30 05:11:53.964631 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:53.964595 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:55.587996 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:55.587960 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:57.004671 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:57.004634 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:57.224183 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:57.224152 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:58.878500 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:58.875281 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:59.091641 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:59.091605 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:59.316083 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:59.316039 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:11:59.535027 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:11:59.535000 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:00.724931 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:00.724872 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:01.067200 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.067060 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg"] Oct 30 05:12:01.067842 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.067814 4793 cpu_manager.go:414] "containerMap: removing container" podUID="11b7ca3f-50a5-4aa2-9b53-241419d21cda" containerName="payload-processing" Oct 30 05:12:01.067842 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.067833 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b7ca3f-50a5-4aa2-9b53-241419d21cda" containerName="payload-processing" Oct 30 05:12:01.068030 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.067885 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3698383d-bb0c-41c5-84d2-1b5aa813151e" containerName="payload-pre-processing" Oct 30 05:12:01.068030 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.067895 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3698383d-bb0c-41c5-84d2-1b5aa813151e" containerName="payload-pre-processing" Oct 30 05:12:01.068030 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.068014 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b7ca3f-50a5-4aa2-9b53-241419d21cda" containerName="payload-processing" Oct 30 05:12:01.068149 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.068035 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9fbe4b-5c65-4e1c-a3f7-6a174131fd2e" containerName="istio-proxy" Oct 30 05:12:01.068149 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.068045 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3698383d-bb0c-41c5-84d2-1b5aa813151e" containerName="payload-pre-processing" Oct 30 05:12:01.072605 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.072589 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.102869 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.102837 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6796b1-openshift-default-dockercfg-h254p\"" Oct 30 05:12:01.108551 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.108522 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg"] Oct 30 05:12:01.142321 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.142298 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d22d10-f91b-48db-ace7-79199901bc24-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.142467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.142327 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.142467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.142356 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.142555 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.142479 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d22d10-f91b-48db-ace7-79199901bc24-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.142555 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.142523 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.142555 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.142542 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.142690 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.142561 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.142733 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.142686 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbjv4\" (UniqueName: \"kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-kube-api-access-qbjv4\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.142733 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.142720 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.244212 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.244180 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.244389 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.244225 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d22d10-f91b-48db-ace7-79199901bc24-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.244389 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.244260 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.244389 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.244279 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.244389 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.244297 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.244389 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.244341 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbjv4\" (UniqueName: \"kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-kube-api-access-qbjv4\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.244389 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.244376 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.244796 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.244564 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d22d10-f91b-48db-ace7-79199901bc24-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.244796 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.244597 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.244796 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.244736 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-data\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.244925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.244832 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.245013 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.244992 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-certs\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.245096 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.245074 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-credential-socket\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.245134 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.245115 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d22d10-f91b-48db-ace7-79199901bc24-istiod-ca-cert\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.246649 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.246631 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-envoy\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.246973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.246955 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d22d10-f91b-48db-ace7-79199901bc24-istio-podinfo\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.272734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.272707 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-istio-token\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.279057 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.279032 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbjv4\" (UniqueName: \"kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-kube-api-access-qbjv4\") pod \"e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.386780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.386690 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:01.719856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.719827 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg"] Oct 30 05:12:01.720335 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:12:01.720312 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d22d10_f91b_48db_ace7_79199901bc24.slice/crio-013711594bd3a739810b059285cbc34e749063fcd379de60c03c35aa96e55ab6 WatchSource:0}: Error finding container 013711594bd3a739810b059285cbc34e749063fcd379de60c03c35aa96e55ab6: Status 404 returned error can't find the container with id 013711594bd3a739810b059285cbc34e749063fcd379de60c03c35aa96e55ab6 Oct 30 05:12:01.723152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.723114 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:12:01.723266 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.723196 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:12:01.723266 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:01.723245 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:12:02.284191 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:02.284150 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:02.534285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:02.534202 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" event={"ID":"79d22d10-f91b-48db-ace7-79199901bc24","Type":"ContainerStarted","Data":"73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573"} Oct 30 05:12:02.534285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:02.534255 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" event={"ID":"79d22d10-f91b-48db-ace7-79199901bc24","Type":"ContainerStarted","Data":"013711594bd3a739810b059285cbc34e749063fcd379de60c03c35aa96e55ab6"} Oct 30 05:12:02.628441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:02.627602 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" podStartSLOduration=1.627578457 podStartE2EDuration="1.627578457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:12:02.593472878 +0000 UTC m=+4445.879347655" watchObservedRunningTime="2026-10-30 05:12:02.627578457 +0000 UTC m=+4445.913453233" Oct 30 05:12:02.911857 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:02.911774 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:03.387886 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:03.387849 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:03.389482 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:03.389450 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 30 05:12:03.389613 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:03.389516 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" podUID="79d22d10-f91b-48db-ace7-79199901bc24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 30 05:12:04.387487 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:04.387429 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 30 05:12:04.387706 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:04.387514 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" podUID="79d22d10-f91b-48db-ace7-79199901bc24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 30 05:12:04.618365 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:04.618328 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:05.304034 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:05.303597 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:05.387330 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:05.387276 4793 patch_prober.go:29] interesting pod/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Oct 30 05:12:05.387590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:05.387343 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" podUID="79d22d10-f91b-48db-ace7-79199901bc24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Oct 30 05:12:05.550981 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:05.550946 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:06.052559 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:06.052522 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:06.146158 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:06.146122 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx"] Oct 30 05:12:06.305602 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:06.305514 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg"] Oct 30 05:12:06.308664 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:06.307811 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:06.309398 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:06.309064 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" podUID="79d22d10-f91b-48db-ace7-79199901bc24" containerName="istio-proxy" containerID="cri-o://73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573" gracePeriod=30 Oct 30 05:12:06.520583 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:06.520543 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:06.990554 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:06.990512 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:07.207343 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:07.207309 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:08.602380 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:08.602343 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:10.195840 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:10.195748 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:11.146860 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.146816 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" podUID="e4a9fa6b-a5b2-407f-87c3-bebc0e0922df" containerName="payload-processing" containerID="cri-o://da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307" gracePeriod=25 Oct 30 05:12:11.220371 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:12:11.220335 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:12:11.424553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.424533 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" Oct 30 05:12:11.519898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.519865 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-plugins-config-volume\") pod \"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df\" (UID: \"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df\") " Oct 30 05:12:11.520057 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.519997 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-kube-api-access-cbcgr\" (UniqueName: \"kubernetes.io/projected/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-kube-api-access-cbcgr\") pod \"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df\" (UID: \"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df\") " Oct 30 05:12:11.520305 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.520276 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-plugins-config-volume" pod "e4a9fa6b-a5b2-407f-87c3-bebc0e0922df" (UID: "e4a9fa6b-a5b2-407f-87c3-bebc0e0922df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:11.520495 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.520475 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:11.522100 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.522079 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-kube-api-access-cbcgr" pod "e4a9fa6b-a5b2-407f-87c3-bebc0e0922df" (UID: "e4a9fa6b-a5b2-407f-87c3-bebc0e0922df"). InnerVolumeSpecName "kube-api-access-cbcgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:11.621856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.621828 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbcgr\" (UniqueName: \"kubernetes.io/projected/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df-kube-api-access-cbcgr\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:11.638023 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.637997 4793 generic.go:356] "Generic (PLEG): container finished" podID="e4a9fa6b-a5b2-407f-87c3-bebc0e0922df" containerID="da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307" exitCode=0 Oct 30 05:12:11.638155 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.638038 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" Oct 30 05:12:11.638155 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.638080 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" event={"ID":"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df","Type":"ContainerDied","Data":"da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307"} Oct 30 05:12:11.638155 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.638129 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx" event={"ID":"e4a9fa6b-a5b2-407f-87c3-bebc0e0922df","Type":"ContainerDied","Data":"f2f890a0639bdf23b2e9d9d0d28454eb21e9349d9f1aeef3cf74f810169c624d"} Oct 30 05:12:11.638155 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.638151 4793 scope.go:122] "RemoveContainer" containerID="da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307" Oct 30 05:12:11.651214 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.651162 4793 scope.go:122] "RemoveContainer" containerID="da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307" Oct 30 05:12:11.651561 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:12:11.651531 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307\": container with ID starting with da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307 not found: ID does not exist" containerID="da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307" Oct 30 05:12:11.651726 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.651573 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307"} err="failed to get container status \"da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307\": rpc error: code = NotFound desc = could not find container \"da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307\": container with ID starting with da73dc3060bbbdf0e21ab06faa0bb68eeac21df30b07d6d45f18662d9a961307 not found: ID does not exist" Oct 30 05:12:11.693485 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.693398 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx"] Oct 30 05:12:11.698692 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.698667 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c0d6e7-7f46967c8b-4vrnx"] Oct 30 05:12:11.805934 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:11.805894 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:12.846261 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:12.846225 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4a9fa6b-a5b2-407f-87c3-bebc0e0922df" path="/var/lib/kubelet/pods/e4a9fa6b-a5b2-407f-87c3-bebc0e0922df/volumes" Oct 30 05:12:13.285032 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.285003 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:13.430012 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.429976 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:13.444930 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.444901 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-kube-api-access-qbjv4\" (UniqueName: \"kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-kube-api-access-qbjv4\") pod \"79d22d10-f91b-48db-ace7-79199901bc24\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " Oct 30 05:12:13.445098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.444942 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-data\") pod \"79d22d10-f91b-48db-ace7-79199901bc24\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " Oct 30 05:12:13.445098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.445088 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79d22d10-f91b-48db-ace7-79199901bc24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d22d10-f91b-48db-ace7-79199901bc24-istio-podinfo\") pod \"79d22d10-f91b-48db-ace7-79199901bc24\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " Oct 30 05:12:13.445220 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.445118 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d22d10-f91b-48db-ace7-79199901bc24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d22d10-f91b-48db-ace7-79199901bc24-istiod-ca-cert\") pod \"79d22d10-f91b-48db-ace7-79199901bc24\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " Oct 30 05:12:13.445276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.445223 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-certs\") pod \"79d22d10-f91b-48db-ace7-79199901bc24\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " Oct 30 05:12:13.445276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.445269 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-credential-socket\") pod \"79d22d10-f91b-48db-ace7-79199901bc24\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " Oct 30 05:12:13.445384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.445302 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-istio-token\" (UniqueName: \"kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-istio-token\") pod \"79d22d10-f91b-48db-ace7-79199901bc24\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " Oct 30 05:12:13.445384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.445299 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-data" pod "79d22d10-f91b-48db-ace7-79199901bc24" (UID: "79d22d10-f91b-48db-ace7-79199901bc24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:13.445523 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.445410 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-envoy\") pod \"79d22d10-f91b-48db-ace7-79199901bc24\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " Oct 30 05:12:13.445523 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.445471 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-socket\") pod \"79d22d10-f91b-48db-ace7-79199901bc24\" (UID: \"79d22d10-f91b-48db-ace7-79199901bc24\") " Oct 30 05:12:13.445633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.445523 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-credential-socket" pod "79d22d10-f91b-48db-ace7-79199901bc24" (UID: "79d22d10-f91b-48db-ace7-79199901bc24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:13.445633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.445567 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d22d10-f91b-48db-ace7-79199901bc24-istiod-ca-cert" pod "79d22d10-f91b-48db-ace7-79199901bc24" (UID: "79d22d10-f91b-48db-ace7-79199901bc24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:13.446211 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.445847 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-certs" pod "79d22d10-f91b-48db-ace7-79199901bc24" (UID: "79d22d10-f91b-48db-ace7-79199901bc24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:13.446211 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.445871 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-socket" pod "79d22d10-f91b-48db-ace7-79199901bc24" (UID: "79d22d10-f91b-48db-ace7-79199901bc24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:13.446211 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.446087 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:13.446211 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.446108 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79d22d10-f91b-48db-ace7-79199901bc24-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:13.446211 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.446123 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:13.446211 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.446135 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:13.446211 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.446149 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:13.447470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.447404 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79d22d10-f91b-48db-ace7-79199901bc24-istio-podinfo" pod "79d22d10-f91b-48db-ace7-79199901bc24" (UID: "79d22d10-f91b-48db-ace7-79199901bc24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:12:13.447588 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.447545 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-kube-api-access-qbjv4" pod "79d22d10-f91b-48db-ace7-79199901bc24" (UID: "79d22d10-f91b-48db-ace7-79199901bc24"). InnerVolumeSpecName "kube-api-access-qbjv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:13.447740 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.447724 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-envoy" pod "79d22d10-f91b-48db-ace7-79199901bc24" (UID: "79d22d10-f91b-48db-ace7-79199901bc24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:13.447881 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.447860 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-istio-token" pod "79d22d10-f91b-48db-ace7-79199901bc24" (UID: "79d22d10-f91b-48db-ace7-79199901bc24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:13.547153 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.547124 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79d22d10-f91b-48db-ace7-79199901bc24-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:13.547153 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.547153 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:13.547153 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.547163 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79d22d10-f91b-48db-ace7-79199901bc24-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:13.547489 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.547172 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbjv4\" (UniqueName: \"kubernetes.io/projected/79d22d10-f91b-48db-ace7-79199901bc24-kube-api-access-qbjv4\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:13.654208 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.654165 4793 generic.go:356] "Generic (PLEG): container finished" podID="79d22d10-f91b-48db-ace7-79199901bc24" containerID="73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573" exitCode=0 Oct 30 05:12:13.654403 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.654230 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" Oct 30 05:12:13.654403 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.654249 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" event={"ID":"79d22d10-f91b-48db-ace7-79199901bc24","Type":"ContainerDied","Data":"73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573"} Oct 30 05:12:13.654403 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.654311 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg" event={"ID":"79d22d10-f91b-48db-ace7-79199901bc24","Type":"ContainerDied","Data":"013711594bd3a739810b059285cbc34e749063fcd379de60c03c35aa96e55ab6"} Oct 30 05:12:13.654403 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.654335 4793 scope.go:122] "RemoveContainer" containerID="73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573" Oct 30 05:12:13.667242 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.667222 4793 scope.go:122] "RemoveContainer" containerID="73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573" Oct 30 05:12:13.667543 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:12:13.667523 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573\": container with ID starting with 73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573 not found: ID does not exist" containerID="73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573" Oct 30 05:12:13.667611 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.667551 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573"} err="failed to get container status \"73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573\": rpc error: code = NotFound desc = could not find container \"73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573\": container with ID starting with 73dee4cb7b689e481e10189ed00972c1f0446992c7bbf9355063fd28848bf573 not found: ID does not exist" Oct 30 05:12:13.737051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.736979 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:13.739346 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.739322 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg"] Oct 30 05:12:13.742827 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:13.742807 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6796b1-openshift-default-56c9bb8b49-df8pg"] Oct 30 05:12:14.188950 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:14.188904 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:14.631361 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:14.631264 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:14.846502 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:14.846468 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d22d10-f91b-48db-ace7-79199901bc24" path="/var/lib/kubelet/pods/79d22d10-f91b-48db-ace7-79199901bc24/volumes" Oct 30 05:12:15.319378 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:15.319340 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:15.541809 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:15.541771 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:17.023083 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:17.023028 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:18.716144 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:18.716105 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:20.405119 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:20.405083 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:22.408109 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:22.408049 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:23.715239 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:23.715202 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:25.375745 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:25.375701 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:27.017889 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:27.017848 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:28.569735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:28.569700 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:28.784381 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:28.784342 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:30.477085 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:30.477044 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:31.621200 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:31.621162 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:31.862223 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:31.862180 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:32.071558 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:32.071521 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:33.425248 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:33.425213 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:34.030296 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:34.030266 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:35.170676 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:35.170635 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:35.395445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:35.390272 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:35.764609 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:35.764568 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:35.994860 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:35.994821 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:36.183327 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:36.183297 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf"] Oct 30 05:12:36.183841 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:36.183517 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" containerID="cri-o://1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63" gracePeriod=30 Oct 30 05:12:36.312434 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:36.312381 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:37.122369 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:37.122326 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:37.532497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:37.532457 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:39.326402 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:39.326362 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:41.016880 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.016843 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:41.220668 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:12:41.220640 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:12:41.472533 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.472510 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 05:12:41.556791 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.556718 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba822078-a0dd-462f-bd7d-add7a00496b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba822078-a0dd-462f-bd7d-add7a00496b1-istiod-ca-cert\") pod \"ba822078-a0dd-462f-bd7d-add7a00496b1\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " Oct 30 05:12:41.556791 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.556778 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-token\" (UniqueName: \"kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-token\") pod \"ba822078-a0dd-462f-bd7d-add7a00496b1\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " Oct 30 05:12:41.556993 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.556894 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-kube-api-access-pg7cn\" (UniqueName: \"kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-kube-api-access-pg7cn\") pod \"ba822078-a0dd-462f-bd7d-add7a00496b1\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " Oct 30 05:12:41.556993 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.556954 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-credential-socket\") pod \"ba822078-a0dd-462f-bd7d-add7a00496b1\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " Oct 30 05:12:41.557104 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.557005 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-certs\") pod \"ba822078-a0dd-462f-bd7d-add7a00496b1\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " Oct 30 05:12:41.557104 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.557045 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-socket\") pod \"ba822078-a0dd-462f-bd7d-add7a00496b1\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " Oct 30 05:12:41.557104 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.557093 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-envoy\") pod \"ba822078-a0dd-462f-bd7d-add7a00496b1\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " Oct 30 05:12:41.557278 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.557131 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba822078-a0dd-462f-bd7d-add7a00496b1-istiod-ca-cert" pod "ba822078-a0dd-462f-bd7d-add7a00496b1" (UID: "ba822078-a0dd-462f-bd7d-add7a00496b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:12:41.557278 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.557141 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-podinfo\") pod \"ba822078-a0dd-462f-bd7d-add7a00496b1\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " Oct 30 05:12:41.557394 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.557293 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-credential-socket" pod "ba822078-a0dd-462f-bd7d-add7a00496b1" (UID: "ba822078-a0dd-462f-bd7d-add7a00496b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:41.557490 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.557439 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-socket" pod "ba822078-a0dd-462f-bd7d-add7a00496b1" (UID: "ba822078-a0dd-462f-bd7d-add7a00496b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:41.557594 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.557561 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-certs" pod "ba822078-a0dd-462f-bd7d-add7a00496b1" (UID: "ba822078-a0dd-462f-bd7d-add7a00496b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:41.557746 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.557724 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:41.557823 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.557751 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:41.557823 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.557761 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:41.557823 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.557769 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba822078-a0dd-462f-bd7d-add7a00496b1-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:41.559608 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.559584 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-podinfo" pod "ba822078-a0dd-462f-bd7d-add7a00496b1" (UID: "ba822078-a0dd-462f-bd7d-add7a00496b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:12:41.559732 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.559616 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-token" pod "ba822078-a0dd-462f-bd7d-add7a00496b1" (UID: "ba822078-a0dd-462f-bd7d-add7a00496b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:41.559732 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.559672 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-kube-api-access-pg7cn" pod "ba822078-a0dd-462f-bd7d-add7a00496b1" (UID: "ba822078-a0dd-462f-bd7d-add7a00496b1"). InnerVolumeSpecName "kube-api-access-pg7cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:12:41.559866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.559753 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-envoy" pod "ba822078-a0dd-462f-bd7d-add7a00496b1" (UID: "ba822078-a0dd-462f-bd7d-add7a00496b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:41.658926 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.658889 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-data\") pod \"ba822078-a0dd-462f-bd7d-add7a00496b1\" (UID: \"ba822078-a0dd-462f-bd7d-add7a00496b1\") " Oct 30 05:12:41.659173 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.659150 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-data" pod "ba822078-a0dd-462f-bd7d-add7a00496b1" (UID: "ba822078-a0dd-462f-bd7d-add7a00496b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:12:41.659534 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.659512 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg7cn\" (UniqueName: \"kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-kube-api-access-pg7cn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:41.659590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.659540 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:41.659590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.659555 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:41.659590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.659566 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:41.659590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.659581 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba822078-a0dd-462f-bd7d-add7a00496b1-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:12:41.856263 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.856175 4793 generic.go:356] "Generic (PLEG): container finished" podID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerID="1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63" exitCode=0 Oct 30 05:12:41.856263 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.856226 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" Oct 30 05:12:41.856507 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.856262 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" event={"ID":"ba822078-a0dd-462f-bd7d-add7a00496b1","Type":"ContainerDied","Data":"1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63"} Oct 30 05:12:41.856507 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.856311 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf" event={"ID":"ba822078-a0dd-462f-bd7d-add7a00496b1","Type":"ContainerDied","Data":"5e1264f9d51c2eb3679c732639495452bea6eefba5a5e2b1ba9a2e41d9d4dcae"} Oct 30 05:12:41.856507 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.856332 4793 scope.go:122] "RemoveContainer" containerID="1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63" Oct 30 05:12:41.869935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.869916 4793 scope.go:122] "RemoveContainer" containerID="59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921" Oct 30 05:12:41.879769 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.879752 4793 scope.go:122] "RemoveContainer" containerID="1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63" Oct 30 05:12:41.880062 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:12:41.880043 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63\": container with ID starting with 1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63 not found: ID does not exist" containerID="1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63" Oct 30 05:12:41.880144 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.880074 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63"} err="failed to get container status \"1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63\": rpc error: code = NotFound desc = could not find container \"1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63\": container with ID starting with 1a5c243b53418f667f05f607424b30159d6971b342cb523c2238434aceecbe63 not found: ID does not exist" Oct 30 05:12:41.880144 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.880100 4793 scope.go:122] "RemoveContainer" containerID="59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921" Oct 30 05:12:41.880360 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:12:41.880338 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921\": container with ID starting with 59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921 not found: ID does not exist" containerID="59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921" Oct 30 05:12:41.880454 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.880370 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921"} err="failed to get container status \"59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921\": rpc error: code = NotFound desc = could not find container \"59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921\": container with ID starting with 59bc042ce2d9d2377e8c3f66171aeaa753e9cd80394c12d2d51310fbcc4e0921 not found: ID does not exist" Oct 30 05:12:41.909446 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.909412 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf"] Oct 30 05:12:41.921042 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:41.921021 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fcf42a-openshift-default-774494965-7n2mf"] Oct 30 05:12:42.701771 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:42.695838 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:42.848469 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:42.848435 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" path="/var/lib/kubelet/pods/ba822078-a0dd-462f-bd7d-add7a00496b1/volumes" Oct 30 05:12:42.909219 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:42.909171 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:43.929905 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:43.929876 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:44.461278 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:44.461235 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:46.102085 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:46.102048 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:47.722642 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:47.722594 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:49.455677 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:49.455633 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:50.076622 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:50.076583 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:51.193743 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:51.193705 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:52.870958 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:52.870915 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:54.497223 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:54.497184 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:56.223035 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:56.222998 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:57.998033 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:57.997992 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:59.270095 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:59.270056 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:12:59.608562 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:12:59.608524 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:01.282965 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:01.282925 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:02.601509 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:02.601471 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9"] Oct 30 05:13:02.602033 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:02.601723 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" podUID="3eaaa85c-519d-4004-8e11-2d46b5dddcfb" containerName="maas-api" containerID="cri-o://a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db" gracePeriod=30 Oct 30 05:13:02.761041 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:02.761003 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:02.931809 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:02.931784 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 05:13:03.021900 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.021864 4793 generic.go:356] "Generic (PLEG): container finished" podID="3eaaa85c-519d-4004-8e11-2d46b5dddcfb" containerID="a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db" exitCode=0 Oct 30 05:13:03.024380 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.022196 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" Oct 30 05:13:03.024380 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.022242 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" event={"ID":"3eaaa85c-519d-4004-8e11-2d46b5dddcfb","Type":"ContainerDied","Data":"a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db"} Oct 30 05:13:03.024380 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.023046 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9" event={"ID":"3eaaa85c-519d-4004-8e11-2d46b5dddcfb","Type":"ContainerDied","Data":"702a9e1b925547a98c012c5d0af1e5760a253f81774f068c7fe319f26fdb8073"} Oct 30 05:13:03.024380 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.023081 4793 scope.go:122] "RemoveContainer" containerID="a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db" Oct 30 05:13:03.052443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.046398 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-openshift-service-ca\") pod \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " Oct 30 05:13:03.052443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.046501 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-maas-api-tls\") pod \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " Oct 30 05:13:03.052443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.046530 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-metrics-certs\") pod \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " Oct 30 05:13:03.052443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.046569 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-kube-api-access-8c7s6\" (UniqueName: \"kubernetes.io/projected/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-kube-api-access-8c7s6\") pod \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\" (UID: \"3eaaa85c-519d-4004-8e11-2d46b5dddcfb\") " Oct 30 05:13:03.052443 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.048858 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-openshift-service-ca" pod "3eaaa85c-519d-4004-8e11-2d46b5dddcfb" (UID: "3eaaa85c-519d-4004-8e11-2d46b5dddcfb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:03.079767 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.074971 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-kube-api-access-8c7s6" pod "3eaaa85c-519d-4004-8e11-2d46b5dddcfb" (UID: "3eaaa85c-519d-4004-8e11-2d46b5dddcfb"). InnerVolumeSpecName "kube-api-access-8c7s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:03.079767 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.075134 4793 scope.go:122] "RemoveContainer" containerID="a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db" Oct 30 05:13:03.079767 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.077917 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-metrics-certs" pod "3eaaa85c-519d-4004-8e11-2d46b5dddcfb" (UID: "3eaaa85c-519d-4004-8e11-2d46b5dddcfb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:13:03.083520 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:13:03.081295 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db\": container with ID starting with a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db not found: ID does not exist" containerID="a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db" Oct 30 05:13:03.083520 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.081341 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db"} err="failed to get container status \"a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db\": rpc error: code = NotFound desc = could not find container \"a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db\": container with ID starting with a596989de459e4c20c2d7cf1e5afc8257e7792cabc1e0060ad08ec0f197130db not found: ID does not exist" Oct 30 05:13:03.085644 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.085590 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-maas-api-tls" pod "3eaaa85c-519d-4004-8e11-2d46b5dddcfb" (UID: "3eaaa85c-519d-4004-8e11-2d46b5dddcfb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:13:03.147866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.147834 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:03.147866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.147863 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:03.147866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.147874 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:03.148119 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.147883 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c7s6\" (UniqueName: \"kubernetes.io/projected/3eaaa85c-519d-4004-8e11-2d46b5dddcfb-kube-api-access-8c7s6\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:03.370893 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.370858 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9"] Oct 30 05:13:03.376751 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.376724 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-4c2415-57647457bc-vc2n9"] Oct 30 05:13:03.451114 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:03.451080 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:04.845991 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:04.845960 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eaaa85c-519d-4004-8e11-2d46b5dddcfb" path="/var/lib/kubelet/pods/3eaaa85c-519d-4004-8e11-2d46b5dddcfb/volumes" Oct 30 05:13:05.113865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:05.113777 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:05.960441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:05.960207 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:06.181440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:06.175696 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:06.624479 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:06.624446 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:06.883695 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:06.883594 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd"] Oct 30 05:13:06.883979 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:06.883893 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" podUID="e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" containerName="istio-proxy" containerID="cri-o://4bc5c4cd4df53a76572b82723afc729b222cdf6cac5f4a522839ce98a279b215" gracePeriod=30 Oct 30 05:13:06.907745 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:06.907709 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:07.118144 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:07.118109 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:07.785188 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:07.785150 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:08.718299 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:08.718262 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:10.512996 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:10.512962 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:11.220398 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:13:11.220364 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:13:12.117479 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.117437 4793 generic.go:356] "Generic (PLEG): container finished" podID="e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" containerID="4bc5c4cd4df53a76572b82723afc729b222cdf6cac5f4a522839ce98a279b215" exitCode=0 Oct 30 05:13:12.117843 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.117512 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" event={"ID":"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c","Type":"ContainerDied","Data":"4bc5c4cd4df53a76572b82723afc729b222cdf6cac5f4a522839ce98a279b215"} Oct 30 05:13:12.179766 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.179704 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 05:13:12.202342 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.202304 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:12.280879 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.280843 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-data\") pod \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " Oct 30 05:13:12.281071 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.280916 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-credential-socket\") pod \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " Oct 30 05:13:12.281071 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.280969 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-certs\") pod \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " Oct 30 05:13:12.281207 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.281094 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-token\" (UniqueName: \"kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-token\") pod \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " Oct 30 05:13:12.281207 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.281131 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-podinfo\") pod \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " Oct 30 05:13:12.281331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.281192 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-credential-socket" pod "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" (UID: "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:12.281331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.281234 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-envoy\") pod \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " Oct 30 05:13:12.281331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.281292 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istiod-ca-cert\") pod \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " Oct 30 05:13:12.281529 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.281338 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-kube-api-access-vlrsq\" (UniqueName: \"kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-kube-api-access-vlrsq\") pod \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " Oct 30 05:13:12.281529 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.281375 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-socket\") pod \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\" (UID: \"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c\") " Oct 30 05:13:12.281529 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.281479 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-certs" pod "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" (UID: "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:12.282482 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.281901 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:12.282482 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.281924 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:12.282482 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.282193 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istiod-ca-cert" pod "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" (UID: "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:12.282748 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.282535 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-socket" pod "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" (UID: "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:12.282748 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.282712 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-data" pod "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" (UID: "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:12.285162 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.285122 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-podinfo" pod "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" (UID: "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:13:12.285384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.285358 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-kube-api-access-vlrsq" pod "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" (UID: "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c"). InnerVolumeSpecName "kube-api-access-vlrsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:12.285788 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.285766 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-token" pod "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" (UID: "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:12.286080 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.286063 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-envoy" pod "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" (UID: "e3e4ad08-7b22-4083-b11a-3ca8c5ab091c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:12.382989 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.382958 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlrsq\" (UniqueName: \"kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-kube-api-access-vlrsq\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:12.382989 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.382987 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:12.382989 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.382997 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:12.383286 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.383005 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:12.383286 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.383013 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:12.383286 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.383021 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:12.383286 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:12.383029 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:13.125242 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:13.125170 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" Oct 30 05:13:13.125242 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:13.125203 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd" event={"ID":"e3e4ad08-7b22-4083-b11a-3ca8c5ab091c","Type":"ContainerDied","Data":"114d4607820150df8dc0d0b5e7fa6d5cb218303a5732f33a26f0d65aab3e3d0a"} Oct 30 05:13:13.125778 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:13.125265 4793 scope.go:122] "RemoveContainer" containerID="4bc5c4cd4df53a76572b82723afc729b222cdf6cac5f4a522839ce98a279b215" Oct 30 05:13:13.189821 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:13.189792 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd"] Oct 30 05:13:13.202202 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:13.202173 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-4c2415-openshift-default-76f7cbd565-kwxxd"] Oct 30 05:13:13.795239 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:13.795196 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:13.804391 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:13.804358 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f"] Oct 30 05:13:13.830512 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:13.830485 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5"] Oct 30 05:13:14.043884 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:14.043847 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:14.263474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:14.263427 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:14.617624 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:14.617531 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:14.846978 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:14.846941 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" path="/var/lib/kubelet/pods/e3e4ad08-7b22-4083-b11a-3ca8c5ab091c/volumes" Oct 30 05:13:15.617894 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:15.617850 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:17.309824 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:17.309790 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:18.805231 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:18.805154 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" podUID="43ca6af4-392d-4c3a-b8de-9cbb133ff81d" containerName="payload-processing" containerID="cri-o://58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d" gracePeriod=25 Oct 30 05:13:18.830758 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:18.830723 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" podUID="47647be1-331a-4103-86f9-fdb6b83d2d9d" containerName="payload-pre-processing" containerID="cri-o://4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda" gracePeriod=25 Oct 30 05:13:19.356603 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:19.356562 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:20.475176 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:20.475138 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:20.694473 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:20.694411 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:22.198110 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:22.198074 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:23.717002 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:23.716961 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" podUID="47647be1-331a-4103-86f9-fdb6b83d2d9d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:13:23.717488 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:23.716965 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" podUID="43ca6af4-392d-4c3a-b8de-9cbb133ff81d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:13:23.782711 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:23.782669 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:25.501948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:25.501898 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:25.989036 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:25.988998 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:27.204868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:27.204820 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:28.723015 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:28.722978 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:28.937913 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:28.937873 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:29.816999 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:29.816965 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:30.271931 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:30.271892 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:30.491346 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:30.491311 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:32.048695 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:32.048656 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:32.789067 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:32.788980 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:33.716471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:33.716409 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" podUID="43ca6af4-392d-4c3a-b8de-9cbb133ff81d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:13:33.716905 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:33.716430 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" podUID="47647be1-331a-4103-86f9-fdb6b83d2d9d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:13:33.842541 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:33.842511 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:34.392870 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:34.392831 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:35.609966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:35.609932 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:35.840474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:35.840411 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:36.342870 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:36.342820 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:37.633723 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:37.633686 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:37.855339 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:37.855304 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:38.070266 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:38.070227 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:38.420765 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:38.420660 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:38.828688 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:38.828524 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:39.111536 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.111504 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" Oct 30 05:13:39.149190 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.149168 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" Oct 30 05:13:39.209445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.207890 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-plugins-config-volume\") pod \"43ca6af4-392d-4c3a-b8de-9cbb133ff81d\" (UID: \"43ca6af4-392d-4c3a-b8de-9cbb133ff81d\") " Oct 30 05:13:39.209445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.208080 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47647be1-331a-4103-86f9-fdb6b83d2d9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47647be1-331a-4103-86f9-fdb6b83d2d9d-plugins-config-volume\") pod \"47647be1-331a-4103-86f9-fdb6b83d2d9d\" (UID: \"47647be1-331a-4103-86f9-fdb6b83d2d9d\") " Oct 30 05:13:39.209445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.208129 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47647be1-331a-4103-86f9-fdb6b83d2d9d-kube-api-access-cxtjt\" (UniqueName: \"kubernetes.io/projected/47647be1-331a-4103-86f9-fdb6b83d2d9d-kube-api-access-cxtjt\") pod \"47647be1-331a-4103-86f9-fdb6b83d2d9d\" (UID: \"47647be1-331a-4103-86f9-fdb6b83d2d9d\") " Oct 30 05:13:39.209445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.208303 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-kube-api-access-77xw4\" (UniqueName: \"kubernetes.io/projected/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-kube-api-access-77xw4\") pod \"43ca6af4-392d-4c3a-b8de-9cbb133ff81d\" (UID: \"43ca6af4-392d-4c3a-b8de-9cbb133ff81d\") " Oct 30 05:13:39.223490 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.220826 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-plugins-config-volume" pod "43ca6af4-392d-4c3a-b8de-9cbb133ff81d" (UID: "43ca6af4-392d-4c3a-b8de-9cbb133ff81d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:39.223490 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.222629 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47647be1-331a-4103-86f9-fdb6b83d2d9d-plugins-config-volume" pod "47647be1-331a-4103-86f9-fdb6b83d2d9d" (UID: "47647be1-331a-4103-86f9-fdb6b83d2d9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:39.272816 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.272757 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-kube-api-access-77xw4" pod "43ca6af4-392d-4c3a-b8de-9cbb133ff81d" (UID: "43ca6af4-392d-4c3a-b8de-9cbb133ff81d"). InnerVolumeSpecName "kube-api-access-77xw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:39.275262 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.275224 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47647be1-331a-4103-86f9-fdb6b83d2d9d-kube-api-access-cxtjt" pod "47647be1-331a-4103-86f9-fdb6b83d2d9d" (UID: "47647be1-331a-4103-86f9-fdb6b83d2d9d"). InnerVolumeSpecName "kube-api-access-cxtjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:39.309685 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.309646 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:39.311172 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.311142 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47647be1-331a-4103-86f9-fdb6b83d2d9d-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:39.311308 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.311176 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxtjt\" (UniqueName: \"kubernetes.io/projected/47647be1-331a-4103-86f9-fdb6b83d2d9d-kube-api-access-cxtjt\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:39.311308 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.311206 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77xw4\" (UniqueName: \"kubernetes.io/projected/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-kube-api-access-77xw4\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:39.311308 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.311223 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43ca6af4-392d-4c3a-b8de-9cbb133ff81d-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:39.387707 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.387623 4793 generic.go:356] "Generic (PLEG): container finished" podID="47647be1-331a-4103-86f9-fdb6b83d2d9d" containerID="4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda" exitCode=0 Oct 30 05:13:39.387707 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.387670 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" Oct 30 05:13:39.387707 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.387692 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" event={"ID":"47647be1-331a-4103-86f9-fdb6b83d2d9d","Type":"ContainerDied","Data":"4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda"} Oct 30 05:13:39.388034 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.387751 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5" event={"ID":"47647be1-331a-4103-86f9-fdb6b83d2d9d","Type":"ContainerDied","Data":"d003cbf62222e66743c3ca49999de5bbab9648c98b5e87eba9572919391b9024"} Oct 30 05:13:39.388034 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.387777 4793 scope.go:122] "RemoveContainer" containerID="4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda" Oct 30 05:13:39.389411 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.389378 4793 generic.go:356] "Generic (PLEG): container finished" podID="43ca6af4-392d-4c3a-b8de-9cbb133ff81d" containerID="58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d" exitCode=0 Oct 30 05:13:39.389570 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.389507 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" event={"ID":"43ca6af4-392d-4c3a-b8de-9cbb133ff81d","Type":"ContainerDied","Data":"58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d"} Oct 30 05:13:39.389570 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.389551 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" event={"ID":"43ca6af4-392d-4c3a-b8de-9cbb133ff81d","Type":"ContainerDied","Data":"cdf931d036bb616e59bfc4a582634c03ea7cb0b2740a5cf9f3456273650dfae6"} Oct 30 05:13:39.389653 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.389589 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f" Oct 30 05:13:39.402097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.402026 4793 scope.go:122] "RemoveContainer" containerID="4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda" Oct 30 05:13:39.402377 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:13:39.402346 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda\": container with ID starting with 4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda not found: ID does not exist" containerID="4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda" Oct 30 05:13:39.402504 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.402384 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda"} err="failed to get container status \"4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda\": rpc error: code = NotFound desc = could not find container \"4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda\": container with ID starting with 4b022e2ae9b65ef80429e5852f09717311dc8a3cc737c48a35b26cb6d4112dda not found: ID does not exist" Oct 30 05:13:39.402504 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.402405 4793 scope.go:122] "RemoveContainer" containerID="58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d" Oct 30 05:13:39.411937 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.411912 4793 scope.go:122] "RemoveContainer" containerID="58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d" Oct 30 05:13:39.412231 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:13:39.412209 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d\": container with ID starting with 58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d not found: ID does not exist" containerID="58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d" Oct 30 05:13:39.412321 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.412237 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d"} err="failed to get container status \"58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d\": rpc error: code = NotFound desc = could not find container \"58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d\": container with ID starting with 58b34d78777781726a8f01976ea83c180736274a3297635e7045ba46076ab50d not found: ID does not exist" Oct 30 05:13:39.440493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.440468 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5"] Oct 30 05:13:39.448619 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.448597 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-bbce72-64f9dd7dd7-n87x5"] Oct 30 05:13:39.485283 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.485227 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f"] Oct 30 05:13:39.489963 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:39.489941 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-bbce72-76dc8f68dd-wqx7f"] Oct 30 05:13:40.035050 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:40.035016 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:40.861338 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:40.861287 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ca6af4-392d-4c3a-b8de-9cbb133ff81d" path="/var/lib/kubelet/pods/43ca6af4-392d-4c3a-b8de-9cbb133ff81d/volumes" Oct 30 05:13:40.862019 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:40.861991 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47647be1-331a-4103-86f9-fdb6b83d2d9d" path="/var/lib/kubelet/pods/47647be1-331a-4103-86f9-fdb6b83d2d9d/volumes" Oct 30 05:13:41.219192 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:13:41.219152 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:13:41.843620 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:41.843579 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:43.413571 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:43.413540 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl"] Oct 30 05:13:43.414024 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:43.413748 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" podUID="a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" containerName="istio-proxy" containerID="cri-o://9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a" gracePeriod=30 Oct 30 05:13:43.425280 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:43.425256 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp"] Oct 30 05:13:43.425482 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:43.425460 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" podUID="a7d234ea-de77-44a4-bec1-0ed26e00a078" containerName="istio-proxy" containerID="cri-o://e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4" gracePeriod=30 Oct 30 05:13:43.428068 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:43.428038 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n"] Oct 30 05:13:43.428233 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:43.428211 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerName="istio-proxy" containerID="cri-o://c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e" gracePeriod=30 Oct 30 05:13:43.436709 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:43.436657 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x"] Oct 30 05:13:43.436891 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:43.436872 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerName="istio-proxy" containerID="cri-o://03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c" gracePeriod=30 Oct 30 05:13:43.445791 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:43.445760 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:43.665722 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:43.665609 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:44.139618 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:44.139578 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc"] Oct 30 05:13:44.329522 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:44.329472 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:44.959929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:44.959889 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:45.203656 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:45.203621 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:46.788727 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:46.788693 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:48.184804 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.184773 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:48.738872 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.738804 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:13:48.836322 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.836282 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-envoy\") pod \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " Oct 30 05:13:48.836507 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.836352 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-socket\") pod \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " Oct 30 05:13:48.836507 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.836393 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-certs\") pod \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " Oct 30 05:13:48.836627 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.836583 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-data\") pod \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " Oct 30 05:13:48.836691 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.836676 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-podinfo\") pod \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " Oct 30 05:13:48.836747 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.836707 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-socket" pod "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" (UID: "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.836747 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.836736 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-token\" (UniqueName: \"kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-token\") pod \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " Oct 30 05:13:48.836854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.836774 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-kube-api-access-q2m95\" (UniqueName: \"kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-kube-api-access-q2m95\") pod \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " Oct 30 05:13:48.836854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.836820 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istiod-ca-cert\") pod \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " Oct 30 05:13:48.836962 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.836882 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-credential-socket\") pod \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\" (UID: \"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e\") " Oct 30 05:13:48.836962 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.836921 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-data" pod "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" (UID: "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.837099 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.837078 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-certs" pod "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" (UID: "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.837288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.837266 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-credential-socket" pod "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" (UID: "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.837569 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.837542 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.837658 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.837598 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.837658 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.837615 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.837658 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.837630 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.839333 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.837919 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istiod-ca-cert" pod "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" (UID: "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:48.841765 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.839798 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-podinfo" pod "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" (UID: "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:13:48.841765 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.840528 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-token" pod "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" (UID: "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:48.841765 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.841314 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 05:13:48.844463 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.842631 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-kube-api-access-q2m95" pod "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" (UID: "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e"). InnerVolumeSpecName "kube-api-access-q2m95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:48.844463 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.843845 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:13:48.846022 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.846002 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:13:48.853597 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.853549 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-envoy" pod "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" (UID: "f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.856097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.856015 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:48.938023 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.937988 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-envoy\") pod \"a7d234ea-de77-44a4-bec1-0ed26e00a078\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " Oct 30 05:13:48.938209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938048 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-credential-socket\") pod \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " Oct 30 05:13:48.938209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938078 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d234ea-de77-44a4-bec1-0ed26e00a078-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d234ea-de77-44a4-bec1-0ed26e00a078-istiod-ca-cert\") pod \"a7d234ea-de77-44a4-bec1-0ed26e00a078\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " Oct 30 05:13:48.938209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938103 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-kube-api-access-fqdlx\" (UniqueName: \"kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-kube-api-access-fqdlx\") pod \"a7d234ea-de77-44a4-bec1-0ed26e00a078\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " Oct 30 05:13:48.938209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938134 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istiod-ca-cert\") pod \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " Oct 30 05:13:48.938209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938172 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-certs\") pod \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " Oct 30 05:13:48.938515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938217 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-certs\") pod \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " Oct 30 05:13:48.938515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938252 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istiod-ca-cert\") pod \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " Oct 30 05:13:48.938515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938271 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-data\") pod \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " Oct 30 05:13:48.938515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938296 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-socket\") pod \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " Oct 30 05:13:48.938515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938349 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-kube-api-access-4rqcw\" (UniqueName: \"kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-kube-api-access-4rqcw\") pod \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " Oct 30 05:13:48.938515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938386 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-data\") pod \"a7d234ea-de77-44a4-bec1-0ed26e00a078\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " Oct 30 05:13:48.938515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938410 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-podinfo\") pod \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " Oct 30 05:13:48.938515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938464 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-envoy\") pod \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " Oct 30 05:13:48.938515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938515 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-credential-socket\") pod \"a7d234ea-de77-44a4-bec1-0ed26e00a078\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938542 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-socket\") pod \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938571 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-certs\") pod \"a7d234ea-de77-44a4-bec1-0ed26e00a078\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938628 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-podinfo\") pod \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938649 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-data" pod "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" (UID: "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938687 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-socket\") pod \"a7d234ea-de77-44a4-bec1-0ed26e00a078\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938716 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-kube-api-access-bcxnq\" (UniqueName: \"kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-kube-api-access-bcxnq\") pod \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938754 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-certs" pod "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" (UID: "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938782 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-token\" (UniqueName: \"kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-token\") pod \"a7d234ea-de77-44a4-bec1-0ed26e00a078\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938810 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-envoy\") pod \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938840 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-data\") pod \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938876 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-token\" (UniqueName: \"kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-token\") pod \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938905 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-credential-socket\") pod \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\" (UID: \"dea2f874-fede-4a13-8cb3-0b17a46c19fd\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938932 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-podinfo\") pod \"a7d234ea-de77-44a4-bec1-0ed26e00a078\" (UID: \"a7d234ea-de77-44a4-bec1-0ed26e00a078\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.938968 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-token\" (UniqueName: \"kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-token\") pod \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\" (UID: \"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1\") " Oct 30 05:13:48.939016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939008 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-certs" pod "dea2f874-fede-4a13-8cb3-0b17a46c19fd" (UID: "dea2f874-fede-4a13-8cb3-0b17a46c19fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.939948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939033 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-credential-socket" pod "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" (UID: "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.939948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939498 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istiod-ca-cert" pod "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" (UID: "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:48.939948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939575 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.939948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939598 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.939948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939613 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.939948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939625 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.939948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939636 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.939948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939648 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.939948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939661 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2m95\" (UniqueName: \"kubernetes.io/projected/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-kube-api-access-q2m95\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.939948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939675 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.939948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939687 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:48.939948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.939817 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-credential-socket" pod "a7d234ea-de77-44a4-bec1-0ed26e00a078" (UID: "a7d234ea-de77-44a4-bec1-0ed26e00a078"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.940894 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.940300 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d234ea-de77-44a4-bec1-0ed26e00a078-istiod-ca-cert" pod "a7d234ea-de77-44a4-bec1-0ed26e00a078" (UID: "a7d234ea-de77-44a4-bec1-0ed26e00a078"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:48.940894 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.940795 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-data" pod "a7d234ea-de77-44a4-bec1-0ed26e00a078" (UID: "a7d234ea-de77-44a4-bec1-0ed26e00a078"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.940894 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.940860 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-socket" pod "a7d234ea-de77-44a4-bec1-0ed26e00a078" (UID: "a7d234ea-de77-44a4-bec1-0ed26e00a078"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.941195 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.941170 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-certs" pod "a7d234ea-de77-44a4-bec1-0ed26e00a078" (UID: "a7d234ea-de77-44a4-bec1-0ed26e00a078"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.941304 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.941282 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-socket" pod "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" (UID: "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.941723 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.941668 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-socket" pod "dea2f874-fede-4a13-8cb3-0b17a46c19fd" (UID: "dea2f874-fede-4a13-8cb3-0b17a46c19fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.942204 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.942182 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istiod-ca-cert" pod "dea2f874-fede-4a13-8cb3-0b17a46c19fd" (UID: "dea2f874-fede-4a13-8cb3-0b17a46c19fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:48.944406 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.942750 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-credential-socket" pod "dea2f874-fede-4a13-8cb3-0b17a46c19fd" (UID: "dea2f874-fede-4a13-8cb3-0b17a46c19fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.944406 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.942769 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-data" pod "dea2f874-fede-4a13-8cb3-0b17a46c19fd" (UID: "dea2f874-fede-4a13-8cb3-0b17a46c19fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.945503 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.945467 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-token" pod "a7d234ea-de77-44a4-bec1-0ed26e00a078" (UID: "a7d234ea-de77-44a4-bec1-0ed26e00a078"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:48.945840 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.945814 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-podinfo" pod "a7d234ea-de77-44a4-bec1-0ed26e00a078" (UID: "a7d234ea-de77-44a4-bec1-0ed26e00a078"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:13:48.947154 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.947127 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-token" pod "dea2f874-fede-4a13-8cb3-0b17a46c19fd" (UID: "dea2f874-fede-4a13-8cb3-0b17a46c19fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:48.947284 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.947173 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-token" pod "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" (UID: "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:48.947370 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.947346 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-kube-api-access-bcxnq" pod "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" (UID: "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1"). InnerVolumeSpecName "kube-api-access-bcxnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:48.948464 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.948044 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-kube-api-access-fqdlx" pod "a7d234ea-de77-44a4-bec1-0ed26e00a078" (UID: "a7d234ea-de77-44a4-bec1-0ed26e00a078"). InnerVolumeSpecName "kube-api-access-fqdlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:48.948464 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.948135 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-envoy" pod "a7d234ea-de77-44a4-bec1-0ed26e00a078" (UID: "a7d234ea-de77-44a4-bec1-0ed26e00a078"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.948815 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.948794 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-podinfo" pod "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" (UID: "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:13:48.949064 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.949031 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-envoy" pod "dea2f874-fede-4a13-8cb3-0b17a46c19fd" (UID: "dea2f874-fede-4a13-8cb3-0b17a46c19fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.949606 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.949382 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-envoy" pod "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" (UID: "a13cc193-7d5e-4a91-a3e2-e00572f2dbd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:48.949606 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.949564 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-podinfo" pod "dea2f874-fede-4a13-8cb3-0b17a46c19fd" (UID: "dea2f874-fede-4a13-8cb3-0b17a46c19fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:13:48.951285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:48.951261 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-kube-api-access-4rqcw" pod "dea2f874-fede-4a13-8cb3-0b17a46c19fd" (UID: "dea2f874-fede-4a13-8cb3-0b17a46c19fd"). InnerVolumeSpecName "kube-api-access-4rqcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:49.041051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.040975 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041003 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041012 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041020 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041028 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041036 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041046 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041054 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041063 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7d234ea-de77-44a4-bec1-0ed26e00a078-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041070 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqdlx\" (UniqueName: \"kubernetes.io/projected/a7d234ea-de77-44a4-bec1-0ed26e00a078-kube-api-access-fqdlx\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041078 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041086 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041094 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041101 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqcw\" (UniqueName: \"kubernetes.io/projected/dea2f874-fede-4a13-8cb3-0b17a46c19fd-kube-api-access-4rqcw\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041110 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041117 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dea2f874-fede-4a13-8cb3-0b17a46c19fd-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041124 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041132 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041141 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dea2f874-fede-4a13-8cb3-0b17a46c19fd-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041149 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041157 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041165 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7d234ea-de77-44a4-bec1-0ed26e00a078-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.041408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.041175 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcxnq\" (UniqueName: \"kubernetes.io/projected/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1-kube-api-access-bcxnq\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:49.406059 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.405965 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:49.477397 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.477364 4793 generic.go:356] "Generic (PLEG): container finished" podID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerID="03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c" exitCode=0 Oct 30 05:13:49.477598 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.477455 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" Oct 30 05:13:49.477598 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.477448 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" event={"ID":"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e","Type":"ContainerDied","Data":"03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c"} Oct 30 05:13:49.477719 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.477623 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x" event={"ID":"f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e","Type":"ContainerDied","Data":"ba65b4d4c8ee8b60e05864f1457e502ed55cb665788843f775c4080e78ee16b6"} Oct 30 05:13:49.477719 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.477661 4793 scope.go:122] "RemoveContainer" containerID="03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c" Oct 30 05:13:49.479619 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.479579 4793 generic.go:356] "Generic (PLEG): container finished" podID="a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" containerID="9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a" exitCode=0 Oct 30 05:13:49.479757 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.479705 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" event={"ID":"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1","Type":"ContainerDied","Data":"9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a"} Oct 30 05:13:49.479757 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.479746 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" event={"ID":"a13cc193-7d5e-4a91-a3e2-e00572f2dbd1","Type":"ContainerDied","Data":"0267324bd02785b5ff32091cad755584c928f91ca65e0f924677d5f4b20c699b"} Oct 30 05:13:49.479899 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.479834 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl" Oct 30 05:13:49.484110 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.484084 4793 generic.go:356] "Generic (PLEG): container finished" podID="a7d234ea-de77-44a4-bec1-0ed26e00a078" containerID="e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4" exitCode=0 Oct 30 05:13:49.484237 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.484148 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" Oct 30 05:13:49.484302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.484262 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" event={"ID":"a7d234ea-de77-44a4-bec1-0ed26e00a078","Type":"ContainerDied","Data":"e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4"} Oct 30 05:13:49.484362 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.484300 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp" event={"ID":"a7d234ea-de77-44a4-bec1-0ed26e00a078","Type":"ContainerDied","Data":"6e44090af1a28d23bdcd949768662b983789ae3bb3617e9f49e3a179f613e2ce"} Oct 30 05:13:49.486032 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.486008 4793 generic.go:356] "Generic (PLEG): container finished" podID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerID="c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e" exitCode=0 Oct 30 05:13:49.486146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.486048 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" event={"ID":"dea2f874-fede-4a13-8cb3-0b17a46c19fd","Type":"ContainerDied","Data":"c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e"} Oct 30 05:13:49.486146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.486074 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" event={"ID":"dea2f874-fede-4a13-8cb3-0b17a46c19fd","Type":"ContainerDied","Data":"5241d3ef116cf0443a7e0fc86a5e4f0689fcda229bfd06d136fc354513f1c5c0"} Oct 30 05:13:49.486146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.486138 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n" Oct 30 05:13:49.492308 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.492292 4793 scope.go:122] "RemoveContainer" containerID="03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c" Oct 30 05:13:49.492869 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:13:49.492843 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c\": container with ID starting with 03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c not found: ID does not exist" containerID="03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c" Oct 30 05:13:49.492946 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.492879 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c"} err="failed to get container status \"03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c\": rpc error: code = NotFound desc = could not find container \"03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c\": container with ID starting with 03f47fa11292ca3d300e97737fe2e9ef7e20c6e69910f70238aab17cca5cb45c not found: ID does not exist" Oct 30 05:13:49.492946 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.492900 4793 scope.go:122] "RemoveContainer" containerID="9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a" Oct 30 05:13:49.503249 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.503231 4793 scope.go:122] "RemoveContainer" containerID="9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a" Oct 30 05:13:49.503540 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:13:49.503514 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a\": container with ID starting with 9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a not found: ID does not exist" containerID="9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a" Oct 30 05:13:49.503609 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.503551 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a"} err="failed to get container status \"9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a\": rpc error: code = NotFound desc = could not find container \"9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a\": container with ID starting with 9f67ebd30dadd9d82511557a4dd06d4436f533139a3561bccce289ad9e9a557a not found: ID does not exist" Oct 30 05:13:49.503609 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.503572 4793 scope.go:122] "RemoveContainer" containerID="e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4" Oct 30 05:13:49.513246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.513229 4793 scope.go:122] "RemoveContainer" containerID="e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4" Oct 30 05:13:49.513519 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:13:49.513500 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4\": container with ID starting with e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4 not found: ID does not exist" containerID="e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4" Oct 30 05:13:49.513609 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.513525 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4"} err="failed to get container status \"e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4\": rpc error: code = NotFound desc = could not find container \"e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4\": container with ID starting with e5220733a1450023e9a30b715a0ce9e55ca7c5cfca7611c1fda9ce12189c1ce4 not found: ID does not exist" Oct 30 05:13:49.513609 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.513548 4793 scope.go:122] "RemoveContainer" containerID="c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e" Oct 30 05:13:49.522780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.522755 4793 scope.go:122] "RemoveContainer" containerID="c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e" Oct 30 05:13:49.523049 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:13:49.523030 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e\": container with ID starting with c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e not found: ID does not exist" containerID="c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e" Oct 30 05:13:49.523107 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.523056 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e"} err="failed to get container status \"c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e\": rpc error: code = NotFound desc = could not find container \"c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e\": container with ID starting with c015693d43a3009721285fa4d523080e05c8be889acac3a12ac6b5369674090e not found: ID does not exist" Oct 30 05:13:49.544760 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.544733 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl"] Oct 30 05:13:49.557412 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.557388 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-78prl"] Oct 30 05:13:49.596727 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.596693 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x"] Oct 30 05:13:49.602615 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.602590 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-vq58x"] Oct 30 05:13:49.632932 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.632905 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:49.636026 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.636000 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp"] Oct 30 05:13:49.640580 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.640560 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-wmnwp"] Oct 30 05:13:49.672685 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.672625 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n"] Oct 30 05:13:49.678876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:49.678853 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-gw78n"] Oct 30 05:13:50.148770 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:50.148734 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:50.433758 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:50.433728 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg"] Oct 30 05:13:50.434234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:50.433988 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" podUID="990dd3c0-bdfb-4779-bf61-df951f964d9f" containerName="istio-proxy" containerID="cri-o://e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f" gracePeriod=30 Oct 30 05:13:50.580043 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:50.580001 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:50.786118 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:50.786035 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:50.847985 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:50.847954 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" path="/var/lib/kubelet/pods/a13cc193-7d5e-4a91-a3e2-e00572f2dbd1/volumes" Oct 30 05:13:50.848447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:50.848430 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d234ea-de77-44a4-bec1-0ed26e00a078" path="/var/lib/kubelet/pods/a7d234ea-de77-44a4-bec1-0ed26e00a078/volumes" Oct 30 05:13:50.848881 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:50.848867 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" path="/var/lib/kubelet/pods/dea2f874-fede-4a13-8cb3-0b17a46c19fd/volumes" Oct 30 05:13:50.849655 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:50.849640 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" path="/var/lib/kubelet/pods/f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e/volumes" Oct 30 05:13:51.211734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:51.211698 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58"] Oct 30 05:13:51.244586 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:51.244550 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7"] Oct 30 05:13:51.451047 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:51.451005 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:51.664022 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:51.663935 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:52.414999 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:52.414964 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:52.632823 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:52.632785 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:52.847910 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:52.847878 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:54.329769 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:54.329734 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:55.196125 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.196082 4793 patch_prober.go:29] interesting pod/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:13:55.196309 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.196159 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" podUID="990dd3c0-bdfb-4779-bf61-df951f964d9f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:13:55.264205 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.264170 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww"] Oct 30 05:13:55.565492 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.565455 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:55.707890 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.707832 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 05:13:55.744883 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.744854 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-credential-socket\") pod \"990dd3c0-bdfb-4779-bf61-df951f964d9f\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " Oct 30 05:13:55.745023 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.744891 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-socket\") pod \"990dd3c0-bdfb-4779-bf61-df951f964d9f\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " Oct 30 05:13:55.745071 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745022 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-kube-api-access-5lmmc\" (UniqueName: \"kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-kube-api-access-5lmmc\") pod \"990dd3c0-bdfb-4779-bf61-df951f964d9f\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " Oct 30 05:13:55.745120 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745066 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-token\" (UniqueName: \"kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-token\") pod \"990dd3c0-bdfb-4779-bf61-df951f964d9f\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " Oct 30 05:13:55.745120 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745111 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-data\") pod \"990dd3c0-bdfb-4779-bf61-df951f964d9f\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " Oct 30 05:13:55.745198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745128 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-credential-socket" pod "990dd3c0-bdfb-4779-bf61-df951f964d9f" (UID: "990dd3c0-bdfb-4779-bf61-df951f964d9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:55.745198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745167 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/990dd3c0-bdfb-4779-bf61-df951f964d9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990dd3c0-bdfb-4779-bf61-df951f964d9f-istiod-ca-cert\") pod \"990dd3c0-bdfb-4779-bf61-df951f964d9f\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " Oct 30 05:13:55.745302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745201 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-podinfo\") pod \"990dd3c0-bdfb-4779-bf61-df951f964d9f\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " Oct 30 05:13:55.745302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745206 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-socket" pod "990dd3c0-bdfb-4779-bf61-df951f964d9f" (UID: "990dd3c0-bdfb-4779-bf61-df951f964d9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:55.745302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745245 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-certs\") pod \"990dd3c0-bdfb-4779-bf61-df951f964d9f\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " Oct 30 05:13:55.745302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745285 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-envoy\") pod \"990dd3c0-bdfb-4779-bf61-df951f964d9f\" (UID: \"990dd3c0-bdfb-4779-bf61-df951f964d9f\") " Oct 30 05:13:55.745578 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745516 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-data" pod "990dd3c0-bdfb-4779-bf61-df951f964d9f" (UID: "990dd3c0-bdfb-4779-bf61-df951f964d9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:55.745643 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745557 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-certs" pod "990dd3c0-bdfb-4779-bf61-df951f964d9f" (UID: "990dd3c0-bdfb-4779-bf61-df951f964d9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:55.745643 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745568 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990dd3c0-bdfb-4779-bf61-df951f964d9f-istiod-ca-cert" pod "990dd3c0-bdfb-4779-bf61-df951f964d9f" (UID: "990dd3c0-bdfb-4779-bf61-df951f964d9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:13:55.745821 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745775 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:55.745821 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745798 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/990dd3c0-bdfb-4779-bf61-df951f964d9f-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:55.745821 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745811 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:55.745934 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745825 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:55.745934 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.745838 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:55.747294 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.747266 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-token" pod "990dd3c0-bdfb-4779-bf61-df951f964d9f" (UID: "990dd3c0-bdfb-4779-bf61-df951f964d9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:55.747412 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.747338 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-podinfo" pod "990dd3c0-bdfb-4779-bf61-df951f964d9f" (UID: "990dd3c0-bdfb-4779-bf61-df951f964d9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:13:55.747675 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.747656 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-kube-api-access-5lmmc" pod "990dd3c0-bdfb-4779-bf61-df951f964d9f" (UID: "990dd3c0-bdfb-4779-bf61-df951f964d9f"). InnerVolumeSpecName "kube-api-access-5lmmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:55.747765 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.747749 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-envoy" pod "990dd3c0-bdfb-4779-bf61-df951f964d9f" (UID: "990dd3c0-bdfb-4779-bf61-df951f964d9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:55.847097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.847064 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lmmc\" (UniqueName: \"kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-kube-api-access-5lmmc\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:55.847097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.847097 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:55.847317 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.847111 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:55.847317 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:55.847123 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/990dd3c0-bdfb-4779-bf61-df951f964d9f-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:56.004470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.004360 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:56.212512 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.212467 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" podUID="44cb423f-1206-4b8d-ad40-f5614e1da7fc" containerName="payload-processing" containerID="cri-o://37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb" gracePeriod=25 Oct 30 05:13:56.245855 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.245815 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" podUID="3c27f602-eb4b-4a29-8b95-ac7b9d736e88" containerName="payload-pre-processing" containerID="cri-o://74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c" gracePeriod=25 Oct 30 05:13:56.304901 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.304818 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:56.538407 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.538373 4793 generic.go:356] "Generic (PLEG): container finished" podID="990dd3c0-bdfb-4779-bf61-df951f964d9f" containerID="e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f" exitCode=0 Oct 30 05:13:56.538593 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.538437 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" Oct 30 05:13:56.538593 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.538444 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" event={"ID":"990dd3c0-bdfb-4779-bf61-df951f964d9f","Type":"ContainerDied","Data":"e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f"} Oct 30 05:13:56.538593 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.538503 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg" event={"ID":"990dd3c0-bdfb-4779-bf61-df951f964d9f","Type":"ContainerDied","Data":"63f81600f48b4e6e9306976601aeed9b74ea1f339dd68467a876948a67d3fcd2"} Oct 30 05:13:56.538593 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.538531 4793 scope.go:122] "RemoveContainer" containerID="e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f" Oct 30 05:13:56.550959 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.550907 4793 scope.go:122] "RemoveContainer" containerID="e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f" Oct 30 05:13:56.551211 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:13:56.551193 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f\": container with ID starting with e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f not found: ID does not exist" containerID="e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f" Oct 30 05:13:56.551302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.551218 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f"} err="failed to get container status \"e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f\": rpc error: code = NotFound desc = could not find container \"e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f\": container with ID starting with e495a6a4f583f8fbc399fb820a64c15eefb04165b3c8abd8ee6681b5cd98615f not found: ID does not exist" Oct 30 05:13:56.602955 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.602892 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg"] Oct 30 05:13:56.614547 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.614521 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c0e964-openshift-default-555bff54df-6q8tg"] Oct 30 05:13:56.850769 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.850712 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990dd3c0-bdfb-4779-bf61-df951f964d9f" path="/var/lib/kubelet/pods/990dd3c0-bdfb-4779-bf61-df951f964d9f/volumes" Oct 30 05:13:56.979548 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:56.977463 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:57.191621 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:57.191587 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:57.770467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:57.770413 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:58.716284 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:58.716241 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:59.161135 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.161036 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" podUID="d1176b48-69f0-47e0-9de0-da5d70421762" containerName="main" containerID="cri-o://de3b9a159091f3e54ec6f2adc6f53d493e5e80f28420236644ff4bb04a8d5f5a" gracePeriod=45 Oct 30 05:13:59.389948 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.389911 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:59.421374 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.421316 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 05:13:59.491168 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.490615 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1176b48-69f0-47e0-9de0-da5d70421762-kube-api-access-n7wfb\" (UniqueName: \"kubernetes.io/projected/d1176b48-69f0-47e0-9de0-da5d70421762-kube-api-access-n7wfb\") pod \"d1176b48-69f0-47e0-9de0-da5d70421762\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " Oct 30 05:13:59.491168 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.490721 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-dshm\") pod \"d1176b48-69f0-47e0-9de0-da5d70421762\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " Oct 30 05:13:59.491168 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.490766 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-tmp-dir\") pod \"d1176b48-69f0-47e0-9de0-da5d70421762\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " Oct 30 05:13:59.491168 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.490822 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-model-cache\") pod \"d1176b48-69f0-47e0-9de0-da5d70421762\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " Oct 30 05:13:59.491168 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.490875 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1176b48-69f0-47e0-9de0-da5d70421762-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1176b48-69f0-47e0-9de0-da5d70421762-tls-certs\") pod \"d1176b48-69f0-47e0-9de0-da5d70421762\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " Oct 30 05:13:59.491168 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.490920 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-home\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-home\") pod \"d1176b48-69f0-47e0-9de0-da5d70421762\" (UID: \"d1176b48-69f0-47e0-9de0-da5d70421762\") " Oct 30 05:13:59.491664 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.491574 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-home" pod "d1176b48-69f0-47e0-9de0-da5d70421762" (UID: "d1176b48-69f0-47e0-9de0-da5d70421762"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:59.492063 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.492036 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-tmp-dir" pod "d1176b48-69f0-47e0-9de0-da5d70421762" (UID: "d1176b48-69f0-47e0-9de0-da5d70421762"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:59.492267 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.492247 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-model-cache" pod "d1176b48-69f0-47e0-9de0-da5d70421762" (UID: "d1176b48-69f0-47e0-9de0-da5d70421762"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:59.494485 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.494449 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1176b48-69f0-47e0-9de0-da5d70421762-kube-api-access-n7wfb" pod "d1176b48-69f0-47e0-9de0-da5d70421762" (UID: "d1176b48-69f0-47e0-9de0-da5d70421762"). InnerVolumeSpecName "kube-api-access-n7wfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:13:59.494931 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.494905 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-dshm" pod "d1176b48-69f0-47e0-9de0-da5d70421762" (UID: "d1176b48-69f0-47e0-9de0-da5d70421762"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:13:59.502442 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.499635 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1176b48-69f0-47e0-9de0-da5d70421762-tls-certs" pod "d1176b48-69f0-47e0-9de0-da5d70421762" (UID: "d1176b48-69f0-47e0-9de0-da5d70421762"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:13:59.540473 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.540445 4793 scope.go:122] "RemoveContainer" containerID="de3b9a159091f3e54ec6f2adc6f53d493e5e80f28420236644ff4bb04a8d5f5a" Oct 30 05:13:59.562526 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.562505 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" Oct 30 05:13:59.562631 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.562540 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" event={"ID":"d1176b48-69f0-47e0-9de0-da5d70421762","Type":"ContainerDied","Data":"de3b9a159091f3e54ec6f2adc6f53d493e5e80f28420236644ff4bb04a8d5f5a"} Oct 30 05:13:59.562631 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.562577 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" event={"ID":"d1176b48-69f0-47e0-9de0-da5d70421762","Type":"ContainerDied","Data":"d9ff136bb07fd202bd407349fb777634e47834ded314cda61792c59f2beaf5cd"} Oct 30 05:13:59.592914 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.592495 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7wfb\" (UniqueName: \"kubernetes.io/projected/d1176b48-69f0-47e0-9de0-da5d70421762-kube-api-access-n7wfb\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:59.592914 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.592528 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:59.592914 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.592546 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:59.592914 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.592560 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:59.592914 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.592573 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1176b48-69f0-47e0-9de0-da5d70421762-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:59.592914 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.592584 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1176b48-69f0-47e0-9de0-da5d70421762-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:13:59.599491 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.599451 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:13:59.612702 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.612680 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc"] Oct 30 05:13:59.618405 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:13:59.618383 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc"] Oct 30 05:14:00.384242 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:00.384195 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-a2f984/simulator-w4-a2f984-kserve-5dd754c5d9-cj2kc" podUID="d1176b48-69f0-47e0-9de0-da5d70421762" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:14:00.844776 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:00.844745 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1176b48-69f0-47e0-9de0-da5d70421762" path="/var/lib/kubelet/pods/d1176b48-69f0-47e0-9de0-da5d70421762/volumes" Oct 30 05:14:01.187403 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:01.187367 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:02.447918 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.447881 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv"] Oct 30 05:14:02.448471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.448130 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" podUID="a19cf987-465a-4521-b6d9-50a9f27a9b7a" containerName="maas-api" containerID="cri-o://92e393dc8cbe2fb08e6a1f78ae9068b111719a2cb2d7d12b9614779c618afa9c" gracePeriod=30 Oct 30 05:14:02.582886 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.582859 4793 generic.go:356] "Generic (PLEG): container finished" podID="a19cf987-465a-4521-b6d9-50a9f27a9b7a" containerID="92e393dc8cbe2fb08e6a1f78ae9068b111719a2cb2d7d12b9614779c618afa9c" exitCode=0 Oct 30 05:14:02.583008 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.582910 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" event={"ID":"a19cf987-465a-4521-b6d9-50a9f27a9b7a","Type":"ContainerDied","Data":"92e393dc8cbe2fb08e6a1f78ae9068b111719a2cb2d7d12b9614779c618afa9c"} Oct 30 05:14:02.600970 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.600942 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:02.708967 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.708904 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 05:14:02.819961 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.819918 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:02.823630 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.823604 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a19cf987-465a-4521-b6d9-50a9f27a9b7a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19cf987-465a-4521-b6d9-50a9f27a9b7a-openshift-service-ca\") pod \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " Oct 30 05:14:02.823842 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.823684 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-maas-api-tls\") pod \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " Oct 30 05:14:02.823842 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.823736 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-metrics-certs\") pod \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " Oct 30 05:14:02.823842 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.823833 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19cf987-465a-4521-b6d9-50a9f27a9b7a-kube-api-access-2qlhv\" (UniqueName: \"kubernetes.io/projected/a19cf987-465a-4521-b6d9-50a9f27a9b7a-kube-api-access-2qlhv\") pod \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\" (UID: \"a19cf987-465a-4521-b6d9-50a9f27a9b7a\") " Oct 30 05:14:02.824003 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.823974 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19cf987-465a-4521-b6d9-50a9f27a9b7a-openshift-service-ca" pod "a19cf987-465a-4521-b6d9-50a9f27a9b7a" (UID: "a19cf987-465a-4521-b6d9-50a9f27a9b7a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:02.824264 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.824238 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19cf987-465a-4521-b6d9-50a9f27a9b7a-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.826061 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.826034 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-maas-api-tls" pod "a19cf987-465a-4521-b6d9-50a9f27a9b7a" (UID: "a19cf987-465a-4521-b6d9-50a9f27a9b7a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:02.826166 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.826079 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19cf987-465a-4521-b6d9-50a9f27a9b7a-kube-api-access-2qlhv" pod "a19cf987-465a-4521-b6d9-50a9f27a9b7a" (UID: "a19cf987-465a-4521-b6d9-50a9f27a9b7a"). InnerVolumeSpecName "kube-api-access-2qlhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:02.826166 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.826131 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-metrics-certs" pod "a19cf987-465a-4521-b6d9-50a9f27a9b7a" (UID: "a19cf987-465a-4521-b6d9-50a9f27a9b7a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:02.925188 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.925155 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qlhv\" (UniqueName: \"kubernetes.io/projected/a19cf987-465a-4521-b6d9-50a9f27a9b7a-kube-api-access-2qlhv\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.925188 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.925184 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.925188 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.925195 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19cf987-465a-4521-b6d9-50a9f27a9b7a-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:02.996959 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.996879 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" podUID="3c27f602-eb4b-4a29-8b95-ac7b9d736e88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:14:02.997198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:02.997172 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" podUID="44cb423f-1206-4b8d-ad40-f5614e1da7fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:14:03.486331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:03.486293 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:03.600358 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:03.600322 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" Oct 30 05:14:03.600565 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:03.600312 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv" event={"ID":"a19cf987-465a-4521-b6d9-50a9f27a9b7a","Type":"ContainerDied","Data":"7836e94465e9bb6200454b35801c7791ba6bf147bee33d31b2b82b7fe13183bb"} Oct 30 05:14:03.600565 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:03.600552 4793 scope.go:122] "RemoveContainer" containerID="92e393dc8cbe2fb08e6a1f78ae9068b111719a2cb2d7d12b9614779c618afa9c" Oct 30 05:14:03.653343 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:03.653318 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv"] Oct 30 05:14:03.665058 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:03.665034 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-05bee8-c7bbd758f-cw6qv"] Oct 30 05:14:04.847339 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:04.847306 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19cf987-465a-4521-b6d9-50a9f27a9b7a" path="/var/lib/kubelet/pods/a19cf987-465a-4521-b6d9-50a9f27a9b7a/volumes" Oct 30 05:14:05.285810 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:05.285770 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:05.497727 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:05.497686 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:07.307919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:07.307883 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:09.111834 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:09.111793 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:09.721805 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:09.721766 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:09.944049 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:09.944014 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:10.284577 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.284531 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" podUID="3d60ccab-2ec3-4c1c-875e-2544990823ed" containerName="main" containerID="cri-o://8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87" gracePeriod=45 Oct 30 05:14:10.438868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.438820 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" podUID="3d60ccab-2ec3-4c1c-875e-2544990823ed" containerName="main" probeResult="failure" output="Get \"https://10.133.0.252:8000/ready\": dial tcp 10.133.0.252:8000: connect: connection refused" Oct 30 05:14:10.532892 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.532846 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:10.540612 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.540560 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:14:10.645707 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.645675 4793 generic.go:356] "Generic (PLEG): container finished" podID="3d60ccab-2ec3-4c1c-875e-2544990823ed" containerID="8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87" exitCode=0 Oct 30 05:14:10.645883 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.645725 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" event={"ID":"3d60ccab-2ec3-4c1c-875e-2544990823ed","Type":"ContainerDied","Data":"8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87"} Oct 30 05:14:10.645883 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.645727 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" Oct 30 05:14:10.645883 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.645754 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww" event={"ID":"3d60ccab-2ec3-4c1c-875e-2544990823ed","Type":"ContainerDied","Data":"aa6beeae1f858b228e1acffdbb817b3af05ca6ddb052a53e78c674f2077993b8"} Oct 30 05:14:10.645883 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.645776 4793 scope.go:122] "RemoveContainer" containerID="8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87" Oct 30 05:14:10.655920 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.655901 4793 scope.go:122] "RemoveContainer" containerID="8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87" Oct 30 05:14:10.656231 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:14:10.656209 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87\": container with ID starting with 8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87 not found: ID does not exist" containerID="8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87" Oct 30 05:14:10.656298 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.656242 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87"} err="failed to get container status \"8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87\": rpc error: code = NotFound desc = could not find container \"8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87\": container with ID starting with 8306eb02fc9c2b8274955abb6491961a749108ed33328547c86e64ed04e9cd87 not found: ID does not exist" Oct 30 05:14:10.709905 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.709877 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-tmp-dir\") pod \"3d60ccab-2ec3-4c1c-875e-2544990823ed\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " Oct 30 05:14:10.710051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.709916 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d60ccab-2ec3-4c1c-875e-2544990823ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/3d60ccab-2ec3-4c1c-875e-2544990823ed-tls-certs\") pod \"3d60ccab-2ec3-4c1c-875e-2544990823ed\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " Oct 30 05:14:10.710051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.709945 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-dshm\") pod \"3d60ccab-2ec3-4c1c-875e-2544990823ed\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " Oct 30 05:14:10.710051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.710011 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-model-cache\") pod \"3d60ccab-2ec3-4c1c-875e-2544990823ed\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " Oct 30 05:14:10.710051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.710041 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d60ccab-2ec3-4c1c-875e-2544990823ed-kube-api-access-v2tdd\" (UniqueName: \"kubernetes.io/projected/3d60ccab-2ec3-4c1c-875e-2544990823ed-kube-api-access-v2tdd\") pod \"3d60ccab-2ec3-4c1c-875e-2544990823ed\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " Oct 30 05:14:10.710278 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.710128 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-home\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-home\") pod \"3d60ccab-2ec3-4c1c-875e-2544990823ed\" (UID: \"3d60ccab-2ec3-4c1c-875e-2544990823ed\") " Oct 30 05:14:10.710278 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.710167 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-tmp-dir" pod "3d60ccab-2ec3-4c1c-875e-2544990823ed" (UID: "3d60ccab-2ec3-4c1c-875e-2544990823ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:10.710410 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.710306 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-model-cache" pod "3d60ccab-2ec3-4c1c-875e-2544990823ed" (UID: "3d60ccab-2ec3-4c1c-875e-2544990823ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:10.710578 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.710556 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-home" pod "3d60ccab-2ec3-4c1c-875e-2544990823ed" (UID: "3d60ccab-2ec3-4c1c-875e-2544990823ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:10.710691 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.710658 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:10.710691 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.710687 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:10.712164 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.712132 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-dshm" pod "3d60ccab-2ec3-4c1c-875e-2544990823ed" (UID: "3d60ccab-2ec3-4c1c-875e-2544990823ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:10.712267 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.712207 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d60ccab-2ec3-4c1c-875e-2544990823ed-tls-certs" pod "3d60ccab-2ec3-4c1c-875e-2544990823ed" (UID: "3d60ccab-2ec3-4c1c-875e-2544990823ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:10.712267 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.712227 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d60ccab-2ec3-4c1c-875e-2544990823ed-kube-api-access-v2tdd" pod "3d60ccab-2ec3-4c1c-875e-2544990823ed" (UID: "3d60ccab-2ec3-4c1c-875e-2544990823ed"). InnerVolumeSpecName "kube-api-access-v2tdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:10.811310 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.811237 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d60ccab-2ec3-4c1c-875e-2544990823ed-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:10.811310 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.811261 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:10.811310 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.811272 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2tdd\" (UniqueName: \"kubernetes.io/projected/3d60ccab-2ec3-4c1c-875e-2544990823ed-kube-api-access-v2tdd\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:10.811310 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.811280 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d60ccab-2ec3-4c1c-875e-2544990823ed-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:10.912920 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.912885 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:10.985355 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.985324 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww"] Oct 30 05:14:10.988979 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:10.988951 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-495ebe/rate-test-model-495ebe-kserve-7d6b4c85d9-42bww"] Oct 30 05:14:11.138497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:11.138397 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:11.218886 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:14:11.218851 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:14:11.840199 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:11.840161 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:12.337167 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.337127 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl"] Oct 30 05:14:12.337433 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.337365 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" podUID="a910f96e-a20b-40a7-9d33-421b71a81f7b" containerName="maas-api" containerID="cri-o://a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0" gracePeriod=30 Oct 30 05:14:12.513710 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.513654 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:12.600730 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.600653 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 05:14:12.661474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.661045 4793 generic.go:356] "Generic (PLEG): container finished" podID="a910f96e-a20b-40a7-9d33-421b71a81f7b" containerID="a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0" exitCode=0 Oct 30 05:14:12.661474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.661085 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" event={"ID":"a910f96e-a20b-40a7-9d33-421b71a81f7b","Type":"ContainerDied","Data":"a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0"} Oct 30 05:14:12.661474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.661099 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" Oct 30 05:14:12.661474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.661148 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl" event={"ID":"a910f96e-a20b-40a7-9d33-421b71a81f7b","Type":"ContainerDied","Data":"4a170c795f668b3f7842782b5fc2e72b70394b36de4265cfb7cf6f6ff6d0fce6"} Oct 30 05:14:12.661474 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.661172 4793 scope.go:122] "RemoveContainer" containerID="a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0" Oct 30 05:14:12.673236 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.673214 4793 scope.go:122] "RemoveContainer" containerID="a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0" Oct 30 05:14:12.673740 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:14:12.673702 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0\": container with ID starting with a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0 not found: ID does not exist" containerID="a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0" Oct 30 05:14:12.673953 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.673747 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0"} err="failed to get container status \"a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0\": rpc error: code = NotFound desc = could not find container \"a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0\": container with ID starting with a1117413174cae31ca85beff33217e4dd45b2dba733a9a4ca49e8c4a68c6cdb0 not found: ID does not exist" Oct 30 05:14:12.729082 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.729053 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-metrics-certs\") pod \"a910f96e-a20b-40a7-9d33-421b71a81f7b\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " Oct 30 05:14:12.729257 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.729128 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a910f96e-a20b-40a7-9d33-421b71a81f7b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a910f96e-a20b-40a7-9d33-421b71a81f7b-openshift-service-ca\") pod \"a910f96e-a20b-40a7-9d33-421b71a81f7b\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " Oct 30 05:14:12.729257 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.729168 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a910f96e-a20b-40a7-9d33-421b71a81f7b-kube-api-access-hkmgh\" (UniqueName: \"kubernetes.io/projected/a910f96e-a20b-40a7-9d33-421b71a81f7b-kube-api-access-hkmgh\") pod \"a910f96e-a20b-40a7-9d33-421b71a81f7b\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " Oct 30 05:14:12.729257 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.729246 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-maas-api-tls\") pod \"a910f96e-a20b-40a7-9d33-421b71a81f7b\" (UID: \"a910f96e-a20b-40a7-9d33-421b71a81f7b\") " Oct 30 05:14:12.729787 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.729659 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a910f96e-a20b-40a7-9d33-421b71a81f7b-openshift-service-ca" pod "a910f96e-a20b-40a7-9d33-421b71a81f7b" (UID: "a910f96e-a20b-40a7-9d33-421b71a81f7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:12.729787 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.729766 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a910f96e-a20b-40a7-9d33-421b71a81f7b-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:12.732753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.732711 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-metrics-certs" pod "a910f96e-a20b-40a7-9d33-421b71a81f7b" (UID: "a910f96e-a20b-40a7-9d33-421b71a81f7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:12.734003 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.733977 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-maas-api-tls" pod "a910f96e-a20b-40a7-9d33-421b71a81f7b" (UID: "a910f96e-a20b-40a7-9d33-421b71a81f7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:12.734144 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.734120 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a910f96e-a20b-40a7-9d33-421b71a81f7b-kube-api-access-hkmgh" pod "a910f96e-a20b-40a7-9d33-421b71a81f7b" (UID: "a910f96e-a20b-40a7-9d33-421b71a81f7b"). InnerVolumeSpecName "kube-api-access-hkmgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:12.831254 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.831224 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:12.831254 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.831251 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkmgh\" (UniqueName: \"kubernetes.io/projected/a910f96e-a20b-40a7-9d33-421b71a81f7b-kube-api-access-hkmgh\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:12.831254 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.831262 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a910f96e-a20b-40a7-9d33-421b71a81f7b-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:12.845905 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.845878 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d60ccab-2ec3-4c1c-875e-2544990823ed" path="/var/lib/kubelet/pods/3d60ccab-2ec3-4c1c-875e-2544990823ed/volumes" Oct 30 05:14:12.888960 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.888932 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:12.999842 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.999803 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" podUID="44cb423f-1206-4b8d-ad40-f5614e1da7fc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:14:13.000025 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:12.999938 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" podUID="3c27f602-eb4b-4a29-8b95-ac7b9d736e88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:14:13.018498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:13.018462 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl"] Oct 30 05:14:13.023798 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:13.023768 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-91705f-7b79fc94c-b6pdl"] Oct 30 05:14:13.517981 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:13.517946 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:14.847981 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:14.847942 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a910f96e-a20b-40a7-9d33-421b71a81f7b" path="/var/lib/kubelet/pods/a910f96e-a20b-40a7-9d33-421b71a81f7b/volumes" Oct 30 05:14:15.212535 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:15.212494 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:16.509262 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.509233 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" Oct 30 05:14:16.533377 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.533355 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" Oct 30 05:14:16.672205 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.672177 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44cb423f-1206-4b8d-ad40-f5614e1da7fc-kube-api-access-h2mm6\" (UniqueName: \"kubernetes.io/projected/44cb423f-1206-4b8d-ad40-f5614e1da7fc-kube-api-access-h2mm6\") pod \"44cb423f-1206-4b8d-ad40-f5614e1da7fc\" (UID: \"44cb423f-1206-4b8d-ad40-f5614e1da7fc\") " Oct 30 05:14:16.672368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.672220 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-plugins-config-volume\") pod \"3c27f602-eb4b-4a29-8b95-ac7b9d736e88\" (UID: \"3c27f602-eb4b-4a29-8b95-ac7b9d736e88\") " Oct 30 05:14:16.672429 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.672403 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-kube-api-access-gf8nj\" (UniqueName: \"kubernetes.io/projected/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-kube-api-access-gf8nj\") pod \"3c27f602-eb4b-4a29-8b95-ac7b9d736e88\" (UID: \"3c27f602-eb4b-4a29-8b95-ac7b9d736e88\") " Oct 30 05:14:16.672475 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.672460 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44cb423f-1206-4b8d-ad40-f5614e1da7fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cb423f-1206-4b8d-ad40-f5614e1da7fc-plugins-config-volume\") pod \"44cb423f-1206-4b8d-ad40-f5614e1da7fc\" (UID: \"44cb423f-1206-4b8d-ad40-f5614e1da7fc\") " Oct 30 05:14:16.672651 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.672627 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-plugins-config-volume" pod "3c27f602-eb4b-4a29-8b95-ac7b9d736e88" (UID: "3c27f602-eb4b-4a29-8b95-ac7b9d736e88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:16.672880 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.672855 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44cb423f-1206-4b8d-ad40-f5614e1da7fc-plugins-config-volume" pod "44cb423f-1206-4b8d-ad40-f5614e1da7fc" (UID: "44cb423f-1206-4b8d-ad40-f5614e1da7fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:16.672880 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.672859 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:16.674402 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.674379 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44cb423f-1206-4b8d-ad40-f5614e1da7fc-kube-api-access-h2mm6" pod "44cb423f-1206-4b8d-ad40-f5614e1da7fc" (UID: "44cb423f-1206-4b8d-ad40-f5614e1da7fc"). InnerVolumeSpecName "kube-api-access-h2mm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:16.674512 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.674455 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-kube-api-access-gf8nj" pod "3c27f602-eb4b-4a29-8b95-ac7b9d736e88" (UID: "3c27f602-eb4b-4a29-8b95-ac7b9d736e88"). InnerVolumeSpecName "kube-api-access-gf8nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:16.687882 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.687854 4793 generic.go:356] "Generic (PLEG): container finished" podID="44cb423f-1206-4b8d-ad40-f5614e1da7fc" containerID="37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb" exitCode=0 Oct 30 05:14:16.687984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.687897 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" Oct 30 05:14:16.687984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.687940 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" event={"ID":"44cb423f-1206-4b8d-ad40-f5614e1da7fc","Type":"ContainerDied","Data":"37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb"} Oct 30 05:14:16.688088 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.687992 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58" event={"ID":"44cb423f-1206-4b8d-ad40-f5614e1da7fc","Type":"ContainerDied","Data":"4fb16e9d0cd7111aa4665ffbe33b9084b370b98cc9cbcd8b1af7fbfebe486030"} Oct 30 05:14:16.688088 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.688017 4793 scope.go:122] "RemoveContainer" containerID="37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb" Oct 30 05:14:16.689660 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.689631 4793 generic.go:356] "Generic (PLEG): container finished" podID="3c27f602-eb4b-4a29-8b95-ac7b9d736e88" containerID="74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c" exitCode=0 Oct 30 05:14:16.689775 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.689665 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" Oct 30 05:14:16.689775 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.689710 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" event={"ID":"3c27f602-eb4b-4a29-8b95-ac7b9d736e88","Type":"ContainerDied","Data":"74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c"} Oct 30 05:14:16.689775 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.689759 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7" event={"ID":"3c27f602-eb4b-4a29-8b95-ac7b9d736e88","Type":"ContainerDied","Data":"116f4f7c72fe26f24f3c8322c272868b6a6eeebc02cfb2d1634cce5f65d4f9bd"} Oct 30 05:14:16.697941 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.697917 4793 scope.go:122] "RemoveContainer" containerID="37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb" Oct 30 05:14:16.698242 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:14:16.698222 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb\": container with ID starting with 37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb not found: ID does not exist" containerID="37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb" Oct 30 05:14:16.698348 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.698248 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb"} err="failed to get container status \"37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb\": rpc error: code = NotFound desc = could not find container \"37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb\": container with ID starting with 37493f6f2c3421d0ace4a7c17a346c096cf798ff60ab11d57ddff66610f548bb not found: ID does not exist" Oct 30 05:14:16.698348 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.698264 4793 scope.go:122] "RemoveContainer" containerID="74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c" Oct 30 05:14:16.706826 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.706808 4793 scope.go:122] "RemoveContainer" containerID="74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c" Oct 30 05:14:16.707083 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:14:16.707064 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c\": container with ID starting with 74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c not found: ID does not exist" containerID="74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c" Oct 30 05:14:16.707141 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.707093 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c"} err="failed to get container status \"74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c\": rpc error: code = NotFound desc = could not find container \"74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c\": container with ID starting with 74f586232cd6bfa2ec84d6966cd3f36c4150ce8d452f26de2bd75c6a6fa3967c not found: ID does not exist" Oct 30 05:14:16.738159 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.738099 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7"] Oct 30 05:14:16.744121 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.744102 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-a2f984-67c675d495-tpzx7"] Oct 30 05:14:16.774246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.774226 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf8nj\" (UniqueName: \"kubernetes.io/projected/3c27f602-eb4b-4a29-8b95-ac7b9d736e88-kube-api-access-gf8nj\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:16.774342 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.774247 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44cb423f-1206-4b8d-ad40-f5614e1da7fc-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:16.774342 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.774258 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2mm6\" (UniqueName: \"kubernetes.io/projected/44cb423f-1206-4b8d-ad40-f5614e1da7fc-kube-api-access-h2mm6\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:16.775207 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.775187 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58"] Oct 30 05:14:16.779317 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.779293 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-a2f984-bd5b789d-ccv58"] Oct 30 05:14:16.851029 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.850948 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c27f602-eb4b-4a29-8b95-ac7b9d736e88" path="/var/lib/kubelet/pods/3c27f602-eb4b-4a29-8b95-ac7b9d736e88/volumes" Oct 30 05:14:16.851674 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.851637 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44cb423f-1206-4b8d-ad40-f5614e1da7fc" path="/var/lib/kubelet/pods/44cb423f-1206-4b8d-ad40-f5614e1da7fc/volumes" Oct 30 05:14:16.898338 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:16.898314 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:18.613242 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:18.613201 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:18.682353 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:18.682323 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs"] Oct 30 05:14:18.853272 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:18.853232 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:19.076683 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:19.076653 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:20.231992 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:20.231954 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:20.862895 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:20.862862 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:21.159721 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:21.159625 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:21.380904 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:21.380865 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:21.596900 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:21.596820 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:22.597640 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:22.597600 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:23.315840 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:23.315802 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:23.533731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:23.533693 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:23.682639 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:23.682602 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" podUID="d4873c9c-f109-4fb1-9653-77b12a7bba1e" containerName="payload-pre-processing" containerID="cri-o://b1a10612c255c1db1be250770c5c7d1cd2f23eb6b270a68fda0826db565f99b5" gracePeriod=25 Oct 30 05:14:24.152538 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:24.152404 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:24.433499 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:24.433465 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:24.655507 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:24.655455 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:24.883365 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:24.883282 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r"] Oct 30 05:14:24.883865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:24.883533 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" containerID="cri-o://eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b" gracePeriod=30 Oct 30 05:14:25.028817 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:25.028781 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:25.249892 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:25.249859 4793 patch_prober.go:29] interesting pod/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:14:25.250058 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:25.249913 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:14:25.469629 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:25.469590 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:25.930192 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:25.930160 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:26.183487 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:26.183372 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:26.295637 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:26.295605 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" podUID="d4873c9c-f109-4fb1-9653-77b12a7bba1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:14:26.408386 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:26.408348 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:26.622249 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:26.622149 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:26.852766 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:26.852734 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:27.098034 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:27.097998 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:27.802277 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:27.802242 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:28.360539 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:28.360500 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:29.387686 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:29.387642 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:29.606341 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:29.606305 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:30.033208 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.033167 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:30.167956 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.167922 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 05:14:30.241718 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.241688 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-certs\") pod \"2da898aa-742b-41a7-9c66-ab97460daf99\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " Oct 30 05:14:30.241885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.241744 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-data\") pod \"2da898aa-742b-41a7-9c66-ab97460daf99\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " Oct 30 05:14:30.241885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.241767 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da898aa-742b-41a7-9c66-ab97460daf99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da898aa-742b-41a7-9c66-ab97460daf99-istiod-ca-cert\") pod \"2da898aa-742b-41a7-9c66-ab97460daf99\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " Oct 30 05:14:30.241885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.241793 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-kube-api-access-hw957\" (UniqueName: \"kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-kube-api-access-hw957\") pod \"2da898aa-742b-41a7-9c66-ab97460daf99\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " Oct 30 05:14:30.241885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.241851 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-istio-token\" (UniqueName: \"kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-istio-token\") pod \"2da898aa-742b-41a7-9c66-ab97460daf99\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " Oct 30 05:14:30.242098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.241903 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-envoy\") pod \"2da898aa-742b-41a7-9c66-ab97460daf99\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " Oct 30 05:14:30.242098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.241971 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-credential-socket\") pod \"2da898aa-742b-41a7-9c66-ab97460daf99\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " Oct 30 05:14:30.242098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.242002 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-certs" pod "2da898aa-742b-41a7-9c66-ab97460daf99" (UID: "2da898aa-742b-41a7-9c66-ab97460daf99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:30.242098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.242021 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-data" pod "2da898aa-742b-41a7-9c66-ab97460daf99" (UID: "2da898aa-742b-41a7-9c66-ab97460daf99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:30.242098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.242061 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-socket\") pod \"2da898aa-742b-41a7-9c66-ab97460daf99\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " Oct 30 05:14:30.242386 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.242106 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2da898aa-742b-41a7-9c66-ab97460daf99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da898aa-742b-41a7-9c66-ab97460daf99-istio-podinfo\") pod \"2da898aa-742b-41a7-9c66-ab97460daf99\" (UID: \"2da898aa-742b-41a7-9c66-ab97460daf99\") " Oct 30 05:14:30.242386 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.242294 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-credential-socket" pod "2da898aa-742b-41a7-9c66-ab97460daf99" (UID: "2da898aa-742b-41a7-9c66-ab97460daf99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:30.242386 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.242324 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da898aa-742b-41a7-9c66-ab97460daf99-istiod-ca-cert" pod "2da898aa-742b-41a7-9c66-ab97460daf99" (UID: "2da898aa-742b-41a7-9c66-ab97460daf99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:30.242586 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.242495 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-socket" pod "2da898aa-742b-41a7-9c66-ab97460daf99" (UID: "2da898aa-742b-41a7-9c66-ab97460daf99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:30.242642 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.242628 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:30.242702 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.242648 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:30.242702 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.242663 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:30.242702 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.242678 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:30.242702 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.242693 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2da898aa-742b-41a7-9c66-ab97460daf99-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:30.244228 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.244202 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2da898aa-742b-41a7-9c66-ab97460daf99-istio-podinfo" pod "2da898aa-742b-41a7-9c66-ab97460daf99" (UID: "2da898aa-742b-41a7-9c66-ab97460daf99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:14:30.244567 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.244545 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-envoy" pod "2da898aa-742b-41a7-9c66-ab97460daf99" (UID: "2da898aa-742b-41a7-9c66-ab97460daf99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:30.244662 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.244615 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-kube-api-access-hw957" pod "2da898aa-742b-41a7-9c66-ab97460daf99" (UID: "2da898aa-742b-41a7-9c66-ab97460daf99"). InnerVolumeSpecName "kube-api-access-hw957". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:30.244725 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.244692 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-istio-token" pod "2da898aa-742b-41a7-9c66-ab97460daf99" (UID: "2da898aa-742b-41a7-9c66-ab97460daf99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:30.344019 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.343990 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw957\" (UniqueName: \"kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-kube-api-access-hw957\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:30.344019 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.344016 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2da898aa-742b-41a7-9c66-ab97460daf99-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:30.344019 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.344025 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2da898aa-742b-41a7-9c66-ab97460daf99-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:30.344228 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.344034 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2da898aa-742b-41a7-9c66-ab97460daf99-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:30.783295 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.783254 4793 generic.go:356] "Generic (PLEG): container finished" podID="2da898aa-742b-41a7-9c66-ab97460daf99" containerID="eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b" exitCode=0 Oct 30 05:14:30.783808 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.783314 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" Oct 30 05:14:30.783808 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.783335 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" event={"ID":"2da898aa-742b-41a7-9c66-ab97460daf99","Type":"ContainerDied","Data":"eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b"} Oct 30 05:14:30.783808 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.783400 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r" event={"ID":"2da898aa-742b-41a7-9c66-ab97460daf99","Type":"ContainerDied","Data":"eb18901bdbd1932d03e1328e979f4b23b8c4605e7af7c21d4b77a2b49283bcdd"} Oct 30 05:14:30.783808 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.783451 4793 scope.go:122] "RemoveContainer" containerID="eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b" Oct 30 05:14:30.794117 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.794097 4793 scope.go:122] "RemoveContainer" containerID="60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9" Oct 30 05:14:30.802447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.802432 4793 scope.go:122] "RemoveContainer" containerID="eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b" Oct 30 05:14:30.802681 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:14:30.802663 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b\": container with ID starting with eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b not found: ID does not exist" containerID="eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b" Oct 30 05:14:30.802747 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.802688 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b"} err="failed to get container status \"eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b\": rpc error: code = NotFound desc = could not find container \"eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b\": container with ID starting with eabfca7662712d646139e2886017d237b16c3fee09f144204116a05e4890872b not found: ID does not exist" Oct 30 05:14:30.802747 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.802708 4793 scope.go:122] "RemoveContainer" containerID="60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9" Oct 30 05:14:30.802936 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:14:30.802919 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9\": container with ID starting with 60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9 not found: ID does not exist" containerID="60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9" Oct 30 05:14:30.802982 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.802941 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9"} err="failed to get container status \"60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9\": rpc error: code = NotFound desc = could not find container \"60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9\": container with ID starting with 60bb1e34073e60b312857ad260400ac0cb365e487023dd5a3aa0ae9ddcf3afd9 not found: ID does not exist" Oct 30 05:14:30.834203 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.834181 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r"] Oct 30 05:14:30.859931 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:30.859903 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-a2f984-openshift-default-c678cb5cc-gwt5r"] Oct 30 05:14:31.760735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:31.760695 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:32.298592 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:32.298557 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:32.536326 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:32.536007 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:32.791528 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:32.791491 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:32.845264 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:32.845231 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" path="/var/lib/kubelet/pods/2da898aa-742b-41a7-9c66-ab97460daf99/volumes" Oct 30 05:14:33.194342 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.194303 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr"] Oct 30 05:14:33.194584 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.194555 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" podUID="721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35" containerName="maas-api" containerID="cri-o://2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84" gracePeriod=30 Oct 30 05:14:33.365190 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.365035 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:33.464050 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.463698 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 05:14:33.579121 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.579088 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-openshift-service-ca\") pod \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " Oct 30 05:14:33.579319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.579129 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-kube-api-access-kjdfh\" (UniqueName: \"kubernetes.io/projected/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-kube-api-access-kjdfh\") pod \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " Oct 30 05:14:33.579319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.579173 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-maas-api-tls\") pod \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " Oct 30 05:14:33.579319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.579197 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-metrics-certs\") pod \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\" (UID: \"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35\") " Oct 30 05:14:33.579583 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.579560 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-openshift-service-ca" pod "721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35" (UID: "721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:33.581401 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.581373 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-metrics-certs" pod "721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35" (UID: "721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:33.581401 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.581373 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-kube-api-access-kjdfh" pod "721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35" (UID: "721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35"). InnerVolumeSpecName "kube-api-access-kjdfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:33.581576 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.581448 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-maas-api-tls" pod "721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35" (UID: "721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:33.680943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.680901 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:33.680943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.680935 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjdfh\" (UniqueName: \"kubernetes.io/projected/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-kube-api-access-kjdfh\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:33.680943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.680949 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:33.681191 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.680963 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:33.757655 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.757571 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9"] Oct 30 05:14:33.805386 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.805353 4793 generic.go:356] "Generic (PLEG): container finished" podID="721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35" containerID="2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84" exitCode=0 Oct 30 05:14:33.805561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.805397 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" Oct 30 05:14:33.805561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.805445 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" event={"ID":"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35","Type":"ContainerDied","Data":"2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84"} Oct 30 05:14:33.805561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.805500 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr" event={"ID":"721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35","Type":"ContainerDied","Data":"313188782be34809d4898c36619e83bd0f6946b0679c80b03513dd36aa53ddec"} Oct 30 05:14:33.805561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.805529 4793 scope.go:122] "RemoveContainer" containerID="2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84" Oct 30 05:14:33.816532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.816514 4793 scope.go:122] "RemoveContainer" containerID="2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84" Oct 30 05:14:33.816814 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:14:33.816794 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84\": container with ID starting with 2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84 not found: ID does not exist" containerID="2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84" Oct 30 05:14:33.816896 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.816825 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84"} err="failed to get container status \"2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84\": rpc error: code = NotFound desc = could not find container \"2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84\": container with ID starting with 2b7836bf096d278e3e6edaa6541b053e9c0fb8fb8603f4ec9b3a1b430555dc84 not found: ID does not exist" Oct 30 05:14:33.861012 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.860979 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr"] Oct 30 05:14:33.865164 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:33.865134 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e6b837-85898b7d49-8dgzr"] Oct 30 05:14:34.014233 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:34.013730 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:34.256876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:34.256839 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:34.844524 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:34.844492 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35" path="/var/lib/kubelet/pods/721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35/volumes" Oct 30 05:14:34.955572 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:34.955540 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:35.716523 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:35.716486 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:36.295882 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:36.295845 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" podUID="d4873c9c-f109-4fb1-9653-77b12a7bba1e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:14:37.312996 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:37.312959 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:38.530166 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:38.530132 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:38.756824 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:38.756791 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:38.758551 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:38.758509 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" podUID="cdc4dabd-cfc2-4a79-aadc-28833d9f99ef" containerName="payload-processing" containerID="cri-o://649a207cb58c46f5079f762937aca98d8736d13a16893424b0d8fd19de5897b1" gracePeriod=25 Oct 30 05:14:39.047540 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:39.047505 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:39.277209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:39.277172 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:39.488396 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:39.488364 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2"] Oct 30 05:14:39.488631 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:39.488601 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" containerID="cri-o://d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411" gracePeriod=30 Oct 30 05:14:39.573097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:39.573059 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:39.906347 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:39.906262 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:40.452700 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:40.452667 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:40.840387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:40.840289 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:41.064059 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:41.064027 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:41.219272 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:14:41.219238 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:14:42.610836 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:42.610784 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:43.816620 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:43.816583 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn"] Oct 30 05:14:43.817057 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:43.816829 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" podUID="b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b" containerName="maas-api" containerID="cri-o://096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81" gracePeriod=30 Oct 30 05:14:43.867928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:43.867896 4793 generic.go:356] "Generic (PLEG): container finished" podID="d4873c9c-f109-4fb1-9653-77b12a7bba1e" containerID="b1a10612c255c1db1be250770c5c7d1cd2f23eb6b270a68fda0826db565f99b5" exitCode=0 Oct 30 05:14:43.868104 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:43.867975 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" event={"ID":"d4873c9c-f109-4fb1-9653-77b12a7bba1e","Type":"ContainerDied","Data":"b1a10612c255c1db1be250770c5c7d1cd2f23eb6b270a68fda0826db565f99b5"} Oct 30 05:14:43.961204 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:43.960949 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" Oct 30 05:14:44.004362 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.004285 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:44.085471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.085412 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 05:14:44.098519 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.098497 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4873c9c-f109-4fb1-9653-77b12a7bba1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4873c9c-f109-4fb1-9653-77b12a7bba1e-plugins-config-volume\") pod \"d4873c9c-f109-4fb1-9653-77b12a7bba1e\" (UID: \"d4873c9c-f109-4fb1-9653-77b12a7bba1e\") " Oct 30 05:14:44.098618 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.098578 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4873c9c-f109-4fb1-9653-77b12a7bba1e-kube-api-access-dhcrp\" (UniqueName: \"kubernetes.io/projected/d4873c9c-f109-4fb1-9653-77b12a7bba1e-kube-api-access-dhcrp\") pod \"d4873c9c-f109-4fb1-9653-77b12a7bba1e\" (UID: \"d4873c9c-f109-4fb1-9653-77b12a7bba1e\") " Oct 30 05:14:44.098930 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.098905 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4873c9c-f109-4fb1-9653-77b12a7bba1e-plugins-config-volume" pod "d4873c9c-f109-4fb1-9653-77b12a7bba1e" (UID: "d4873c9c-f109-4fb1-9653-77b12a7bba1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:44.101089 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.101060 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4873c9c-f109-4fb1-9653-77b12a7bba1e-kube-api-access-dhcrp" pod "d4873c9c-f109-4fb1-9653-77b12a7bba1e" (UID: "d4873c9c-f109-4fb1-9653-77b12a7bba1e"). InnerVolumeSpecName "kube-api-access-dhcrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:44.199863 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.199780 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-metrics-certs\") pod \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " Oct 30 05:14:44.200042 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.199870 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-maas-api-tls\") pod \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " Oct 30 05:14:44.200042 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.199922 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-kube-api-access-jqtk6\" (UniqueName: \"kubernetes.io/projected/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-kube-api-access-jqtk6\") pod \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " Oct 30 05:14:44.200042 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.199997 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-openshift-service-ca\") pod \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\" (UID: \"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b\") " Oct 30 05:14:44.200505 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.200471 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-openshift-service-ca" pod "b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b" (UID: "b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:44.200712 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.200677 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhcrp\" (UniqueName: \"kubernetes.io/projected/d4873c9c-f109-4fb1-9653-77b12a7bba1e-kube-api-access-dhcrp\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:44.200712 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.200702 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:44.200926 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.200717 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4873c9c-f109-4fb1-9653-77b12a7bba1e-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:44.201963 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.201940 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-metrics-certs" pod "b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b" (UID: "b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:44.202066 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.202008 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-maas-api-tls" pod "b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b" (UID: "b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:14:44.202357 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.202342 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-kube-api-access-jqtk6" pod "b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b" (UID: "b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b"). InnerVolumeSpecName "kube-api-access-jqtk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:44.302190 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.302157 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:44.302190 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.302187 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:44.302190 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.302196 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqtk6\" (UniqueName: \"kubernetes.io/projected/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b-kube-api-access-jqtk6\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:44.640335 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.640213 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:44.758393 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.758323 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 05:14:44.875413 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.875385 4793 generic.go:356] "Generic (PLEG): container finished" podID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerID="d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411" exitCode=0 Oct 30 05:14:44.875819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.875446 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" Oct 30 05:14:44.875819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.875445 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" event={"ID":"3abcfdfb-40e7-441d-bff1-c74858fe5aaf","Type":"ContainerDied","Data":"d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411"} Oct 30 05:14:44.875819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.875597 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2" event={"ID":"3abcfdfb-40e7-441d-bff1-c74858fe5aaf","Type":"ContainerDied","Data":"0729b28c3d29589ef526628a404a4dbccdb2835bba78a3757a5ed58aff60a60d"} Oct 30 05:14:44.875819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.875646 4793 scope.go:122] "RemoveContainer" containerID="d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411" Oct 30 05:14:44.876935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.876876 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" Oct 30 05:14:44.876935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.876925 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs" event={"ID":"d4873c9c-f109-4fb1-9653-77b12a7bba1e","Type":"ContainerDied","Data":"346e71bd586b378b7947b813c7030a2314d4a5ed0d4ae2b31c9008784bf5d684"} Oct 30 05:14:44.878494 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.878470 4793 generic.go:356] "Generic (PLEG): container finished" podID="b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b" containerID="096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81" exitCode=0 Oct 30 05:14:44.878621 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.878521 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" event={"ID":"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b","Type":"ContainerDied","Data":"096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81"} Oct 30 05:14:44.878621 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.878532 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" Oct 30 05:14:44.878621 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.878558 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn" event={"ID":"b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b","Type":"ContainerDied","Data":"3eebb4a667442c49481b6672e5e3e6d3b9d78ac233c53dbcb037210712edb51a"} Oct 30 05:14:44.886720 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.886703 4793 scope.go:122] "RemoveContainer" containerID="5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e" Oct 30 05:14:44.897659 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.897638 4793 scope.go:122] "RemoveContainer" containerID="d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411" Oct 30 05:14:44.897918 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:14:44.897895 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411\": container with ID starting with d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411 not found: ID does not exist" containerID="d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411" Oct 30 05:14:44.897989 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.897929 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411"} err="failed to get container status \"d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411\": rpc error: code = NotFound desc = could not find container \"d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411\": container with ID starting with d385f4db8dbf46b685cf4ac5d8e21ae1837d9483531077c8748574a089886411 not found: ID does not exist" Oct 30 05:14:44.897989 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.897965 4793 scope.go:122] "RemoveContainer" containerID="5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e" Oct 30 05:14:44.898233 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:14:44.898215 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e\": container with ID starting with 5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e not found: ID does not exist" containerID="5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e" Oct 30 05:14:44.898296 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.898243 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e"} err="failed to get container status \"5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e\": rpc error: code = NotFound desc = could not find container \"5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e\": container with ID starting with 5a607e957ecef28f5f6f6b4df0545cfca3c187048ef62bc482ad82f93550174e not found: ID does not exist" Oct 30 05:14:44.898296 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.898264 4793 scope.go:122] "RemoveContainer" containerID="b1a10612c255c1db1be250770c5c7d1cd2f23eb6b270a68fda0826db565f99b5" Oct 30 05:14:44.906609 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.906591 4793 scope.go:122] "RemoveContainer" containerID="096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81" Oct 30 05:14:44.908805 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.908783 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-socket\") pod \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " Oct 30 05:14:44.908909 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.908817 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-token\" (UniqueName: \"kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-token\") pod \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " Oct 30 05:14:44.908909 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.908853 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-podinfo\") pod \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " Oct 30 05:14:44.908909 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.908880 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-envoy\") pod \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " Oct 30 05:14:44.909102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.908926 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-kube-api-access-jq48f\" (UniqueName: \"kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-kube-api-access-jq48f\") pod \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " Oct 30 05:14:44.909102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.908958 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istiod-ca-cert\") pod \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " Oct 30 05:14:44.909102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.908993 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-certs\") pod \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " Oct 30 05:14:44.909102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.909044 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-socket" pod "3abcfdfb-40e7-441d-bff1-c74858fe5aaf" (UID: "3abcfdfb-40e7-441d-bff1-c74858fe5aaf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:44.909313 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.909145 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-credential-socket\") pod \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " Oct 30 05:14:44.909313 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.909196 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-data\") pod \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\" (UID: \"3abcfdfb-40e7-441d-bff1-c74858fe5aaf\") " Oct 30 05:14:44.909532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.909501 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-credential-socket" pod "3abcfdfb-40e7-441d-bff1-c74858fe5aaf" (UID: "3abcfdfb-40e7-441d-bff1-c74858fe5aaf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:44.909740 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.909702 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-data" pod "3abcfdfb-40e7-441d-bff1-c74858fe5aaf" (UID: "3abcfdfb-40e7-441d-bff1-c74858fe5aaf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:44.909877 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.909852 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-certs" pod "3abcfdfb-40e7-441d-bff1-c74858fe5aaf" (UID: "3abcfdfb-40e7-441d-bff1-c74858fe5aaf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:44.909983 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.909905 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istiod-ca-cert" pod "3abcfdfb-40e7-441d-bff1-c74858fe5aaf" (UID: "3abcfdfb-40e7-441d-bff1-c74858fe5aaf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:44.910075 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.909910 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:44.910191 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.910171 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:44.910310 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.910194 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:44.911886 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.911859 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-token" pod "3abcfdfb-40e7-441d-bff1-c74858fe5aaf" (UID: "3abcfdfb-40e7-441d-bff1-c74858fe5aaf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:44.911987 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.911937 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-envoy" pod "3abcfdfb-40e7-441d-bff1-c74858fe5aaf" (UID: "3abcfdfb-40e7-441d-bff1-c74858fe5aaf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:14:44.912044 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.912025 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-podinfo" pod "3abcfdfb-40e7-441d-bff1-c74858fe5aaf" (UID: "3abcfdfb-40e7-441d-bff1-c74858fe5aaf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:14:44.912772 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.912754 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-kube-api-access-jq48f" pod "3abcfdfb-40e7-441d-bff1-c74858fe5aaf" (UID: "3abcfdfb-40e7-441d-bff1-c74858fe5aaf"). InnerVolumeSpecName "kube-api-access-jq48f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:44.916801 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.916771 4793 scope.go:122] "RemoveContainer" containerID="096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81" Oct 30 05:14:44.917107 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:14:44.917087 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81\": container with ID starting with 096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81 not found: ID does not exist" containerID="096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81" Oct 30 05:14:44.917174 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.917114 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81"} err="failed to get container status \"096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81\": rpc error: code = NotFound desc = could not find container \"096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81\": container with ID starting with 096e0e1e1241e5d6df69ac399da3a61717b377d5029326d5bccb9a6260048b81 not found: ID does not exist" Oct 30 05:14:44.926210 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.926188 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn"] Oct 30 05:14:44.930627 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.930605 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-e070ea-549788df65-w49sn"] Oct 30 05:14:44.964218 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.964192 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs"] Oct 30 05:14:44.968659 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:44.968634 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-b195d0-c454c77b7-jmtxs"] Oct 30 05:14:45.011398 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:45.011329 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:45.011548 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:45.011443 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:45.011548 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:45.011481 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:45.011548 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:45.011498 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq48f\" (UniqueName: \"kubernetes.io/projected/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-kube-api-access-jq48f\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:45.011548 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:45.011513 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:45.011548 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:45.011527 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3abcfdfb-40e7-441d-bff1-c74858fe5aaf-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:45.220646 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:45.220617 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2"] Oct 30 05:14:45.226375 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:45.226350 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e6b837-openshift-default-77d9944fd9-j2fj2"] Oct 30 05:14:46.348241 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:46.348200 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:46.584487 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:46.584447 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:46.802641 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:46.802600 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:46.846553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:46.846518 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" path="/var/lib/kubelet/pods/3abcfdfb-40e7-441d-bff1-c74858fe5aaf/volumes" Oct 30 05:14:46.847128 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:46.847110 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b" path="/var/lib/kubelet/pods/b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b/volumes" Oct 30 05:14:46.847660 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:46.847635 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4873c9c-f109-4fb1-9653-77b12a7bba1e" path="/var/lib/kubelet/pods/d4873c9c-f109-4fb1-9653-77b12a7bba1e/volumes" Oct 30 05:14:47.392441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:47.392379 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:47.752059 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:47.752021 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:48.006687 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:48.006571 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:48.225479 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:48.225402 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:48.576913 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:48.576880 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:48.620440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:48.620377 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" podUID="cdc4dabd-cfc2-4a79-aadc-28833d9f99ef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:14:48.836241 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:48.836150 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:49.624813 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:49.624769 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:49.946734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:49.946698 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:51.622481 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:51.622441 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:53.220334 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:53.220298 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:53.458378 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:53.458339 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:54.713466 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:54.713406 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:55.226191 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:55.226157 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:56.936848 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:56.936817 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:57.148827 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:57.148790 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:58.620989 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:58.620951 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" podUID="cdc4dabd-cfc2-4a79-aadc-28833d9f99ef" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:14:58.697787 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:58.697743 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:58.971408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:58.971370 4793 generic.go:356] "Generic (PLEG): container finished" podID="cdc4dabd-cfc2-4a79-aadc-28833d9f99ef" containerID="649a207cb58c46f5079f762937aca98d8736d13a16893424b0d8fd19de5897b1" exitCode=0 Oct 30 05:14:58.971611 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:58.971471 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" event={"ID":"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef","Type":"ContainerDied","Data":"649a207cb58c46f5079f762937aca98d8736d13a16893424b0d8fd19de5897b1"} Oct 30 05:14:59.025326 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.025302 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" Oct 30 05:14:59.160380 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.160308 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-kube-api-access-cgb8n\" (UniqueName: \"kubernetes.io/projected/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-kube-api-access-cgb8n\") pod \"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef\" (UID: \"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef\") " Oct 30 05:14:59.160380 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.160359 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-plugins-config-volume\") pod \"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef\" (UID: \"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef\") " Oct 30 05:14:59.161469 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.161437 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-plugins-config-volume" pod "cdc4dabd-cfc2-4a79-aadc-28833d9f99ef" (UID: "cdc4dabd-cfc2-4a79-aadc-28833d9f99ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:14:59.163860 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.163828 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-kube-api-access-cgb8n" pod "cdc4dabd-cfc2-4a79-aadc-28833d9f99ef" (UID: "cdc4dabd-cfc2-4a79-aadc-28833d9f99ef"). InnerVolumeSpecName "kube-api-access-cgb8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:14:59.261487 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.261452 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgb8n\" (UniqueName: \"kubernetes.io/projected/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-kube-api-access-cgb8n\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:59.261487 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.261480 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:14:59.512969 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.512937 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:59.610630 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.610594 4793 scope.go:122] "RemoveContainer" containerID="649a207cb58c46f5079f762937aca98d8736d13a16893424b0d8fd19de5897b1" Oct 30 05:14:59.749497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.749456 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:59.965996 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.965960 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:14:59.981594 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.981551 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" event={"ID":"cdc4dabd-cfc2-4a79-aadc-28833d9f99ef","Type":"ContainerDied","Data":"37712b97f2e2240dcf89534d46ede8e6f1c8d6a157cc2b2e23f6fb8122300d0b"} Oct 30 05:14:59.981802 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:14:59.981628 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9" Oct 30 05:15:00.058595 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.058564 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9"] Oct 30 05:15:00.062360 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.062334 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-e070ea-b75f674b-hr9p9"] Oct 30 05:15:00.159648 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.159615 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s"] Oct 30 05:15:00.160032 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160017 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3eaaa85c-519d-4004-8e11-2d46b5dddcfb" containerName="maas-api" Oct 30 05:15:00.160097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160032 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eaaa85c-519d-4004-8e11-2d46b5dddcfb" containerName="maas-api" Oct 30 05:15:00.160097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160051 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" containerName="istio-proxy" Oct 30 05:15:00.160097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160056 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" containerName="istio-proxy" Oct 30 05:15:00.160097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160065 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d4873c9c-f109-4fb1-9653-77b12a7bba1e" containerName="payload-pre-processing" Oct 30 05:15:00.160097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160070 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4873c9c-f109-4fb1-9653-77b12a7bba1e" containerName="payload-pre-processing" Oct 30 05:15:00.160097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160080 4793 cpu_manager.go:414] "containerMap: removing container" podUID="b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b" containerName="maas-api" Oct 30 05:15:00.160097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160086 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b" containerName="maas-api" Oct 30 05:15:00.160097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160096 4793 cpu_manager.go:414] "containerMap: removing container" podUID="cdc4dabd-cfc2-4a79-aadc-28833d9f99ef" containerName="payload-processing" Oct 30 05:15:00.160097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160102 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc4dabd-cfc2-4a79-aadc-28833d9f99ef" containerName="payload-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160108 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160113 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160119 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160124 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160131 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160136 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160146 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a910f96e-a20b-40a7-9d33-421b71a81f7b" containerName="maas-api" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160150 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a910f96e-a20b-40a7-9d33-421b71a81f7b" containerName="maas-api" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160155 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160159 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160165 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160170 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160178 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160182 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160191 4793 cpu_manager.go:414] "containerMap: removing container" podUID="44cb423f-1206-4b8d-ad40-f5614e1da7fc" containerName="payload-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160195 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44cb423f-1206-4b8d-ad40-f5614e1da7fc" containerName="payload-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160202 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a7d234ea-de77-44a4-bec1-0ed26e00a078" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160207 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d234ea-de77-44a4-bec1-0ed26e00a078" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160214 4793 cpu_manager.go:414] "containerMap: removing container" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160219 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160226 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d1176b48-69f0-47e0-9de0-da5d70421762" containerName="main" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160231 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1176b48-69f0-47e0-9de0-da5d70421762" containerName="main" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160237 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3c27f602-eb4b-4a29-8b95-ac7b9d736e88" containerName="payload-pre-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160242 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c27f602-eb4b-4a29-8b95-ac7b9d736e88" containerName="payload-pre-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160253 4793 cpu_manager.go:414] "containerMap: removing container" podUID="79d22d10-f91b-48db-ace7-79199901bc24" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160258 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d22d10-f91b-48db-ace7-79199901bc24" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160263 4793 cpu_manager.go:414] "containerMap: removing container" podUID="e4a9fa6b-a5b2-407f-87c3-bebc0e0922df" containerName="payload-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160268 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a9fa6b-a5b2-407f-87c3-bebc0e0922df" containerName="payload-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160274 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160278 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160285 4793 cpu_manager.go:414] "containerMap: removing container" podUID="43ca6af4-392d-4c3a-b8de-9cbb133ff81d" containerName="payload-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160289 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ca6af4-392d-4c3a-b8de-9cbb133ff81d" containerName="payload-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160296 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a19cf987-465a-4521-b6d9-50a9f27a9b7a" containerName="maas-api" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160300 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19cf987-465a-4521-b6d9-50a9f27a9b7a" containerName="maas-api" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160310 4793 cpu_manager.go:414] "containerMap: removing container" podUID="47647be1-331a-4103-86f9-fdb6b83d2d9d" containerName="payload-pre-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160315 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47647be1-331a-4103-86f9-fdb6b83d2d9d" containerName="payload-pre-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160321 4793 cpu_manager.go:414] "containerMap: removing container" podUID="3d60ccab-2ec3-4c1c-875e-2544990823ed" containerName="main" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160326 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d60ccab-2ec3-4c1c-875e-2544990823ed" containerName="main" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160333 4793 cpu_manager.go:414] "containerMap: removing container" podUID="721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35" containerName="maas-api" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160337 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35" containerName="maas-api" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160344 4793 cpu_manager.go:414] "containerMap: removing container" podUID="990dd3c0-bdfb-4779-bf61-df951f964d9f" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160348 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990dd3c0-bdfb-4779-bf61-df951f964d9f" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160355 4793 cpu_manager.go:414] "containerMap: removing container" podUID="e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160359 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160430 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3e4ad08-7b22-4083-b11a-3ca8c5ab091c" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160439 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eaaa85c-519d-4004-8e11-2d46b5dddcfb" containerName="maas-api" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160446 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13cc193-7d5e-4a91-a3e2-e00572f2dbd1" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160453 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4873c9c-f109-4fb1-9653-77b12a7bba1e" containerName="payload-pre-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160460 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7a9e7c7-6603-4e72-9159-0a2cc7d5f04b" containerName="maas-api" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160466 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc4dabd-cfc2-4a79-aadc-28833d9f99ef" containerName="payload-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160472 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160476 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da898aa-742b-41a7-9c66-ab97460daf99" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160482 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160487 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a910f96e-a20b-40a7-9d33-421b71a81f7b" containerName="maas-api" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160493 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160498 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3abcfdfb-40e7-441d-bff1-c74858fe5aaf" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160503 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4497e6f-eb1b-4b73-9bf8-c0de4bf5102e" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160508 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="44cb423f-1206-4b8d-ad40-f5614e1da7fc" containerName="payload-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160514 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d234ea-de77-44a4-bec1-0ed26e00a078" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160519 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea2f874-fede-4a13-8cb3-0b17a46c19fd" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160523 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1176b48-69f0-47e0-9de0-da5d70421762" containerName="main" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160528 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c27f602-eb4b-4a29-8b95-ac7b9d736e88" containerName="payload-pre-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160535 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d22d10-f91b-48db-ace7-79199901bc24" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160541 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a9fa6b-a5b2-407f-87c3-bebc0e0922df" containerName="payload-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160547 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba822078-a0dd-462f-bd7d-add7a00496b1" containerName="istio-proxy" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160552 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ca6af4-392d-4c3a-b8de-9cbb133ff81d" containerName="payload-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160557 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19cf987-465a-4521-b6d9-50a9f27a9b7a" containerName="maas-api" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160562 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="47647be1-331a-4103-86f9-fdb6b83d2d9d" containerName="payload-pre-processing" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160568 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d60ccab-2ec3-4c1c-875e-2544990823ed" containerName="main" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160572 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="721e8c36-82b0-4a8c-8f0e-5a4ad1b77d35" containerName="maas-api" Oct 30 05:15:00.160633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.160576 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="990dd3c0-bdfb-4779-bf61-df951f964d9f" containerName="istio-proxy" Oct 30 05:15:00.165601 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.165581 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz"] Oct 30 05:15:00.165725 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.165691 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" Oct 30 05:15:00.168848 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.168824 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2"] Oct 30 05:15:00.168971 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.168952 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" Oct 30 05:15:00.172184 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.172163 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" Oct 30 05:15:00.179845 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.179824 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s"] Oct 30 05:15:00.182398 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.182347 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:15:00.182563 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.182528 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:15:00.182685 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.182618 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-pt6g2\"" Oct 30 05:15:00.190070 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.190052 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz"] Oct 30 05:15:00.217989 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.217930 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2"] Oct 30 05:15:00.271762 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.271735 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48947e23-be16-44b9-9c61-209679df23a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2\" (UID: \"48947e23-be16-44b9-9c61-209679df23a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" Oct 30 05:15:00.271906 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.271768 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzhwh\" (UniqueName: \"kubernetes.io/projected/48947e23-be16-44b9-9c61-209679df23a9-kube-api-access-gzhwh\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2\" (UID: \"48947e23-be16-44b9-9c61-209679df23a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" Oct 30 05:15:00.271906 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.271872 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvlqg\" (UniqueName: \"kubernetes.io/projected/4259a987-fbcc-4edf-867e-0f21fff10f9f-kube-api-access-rvlqg\") pod \"maas-api-key-cleanup-29888955-sp49s\" (UID: \"4259a987-fbcc-4edf-867e-0f21fff10f9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" Oct 30 05:15:00.271906 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.271895 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4259a987-fbcc-4edf-867e-0f21fff10f9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888955-sp49s\" (UID: \"4259a987-fbcc-4edf-867e-0f21fff10f9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" Oct 30 05:15:00.272038 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.271923 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbb2w\" (UniqueName: \"kubernetes.io/projected/1c630cf4-f669-4563-8483-f9f367841185-kube-api-access-dbb2w\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz\" (UID: \"1c630cf4-f669-4563-8483-f9f367841185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" Oct 30 05:15:00.272038 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.271994 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c630cf4-f669-4563-8483-f9f367841185-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz\" (UID: \"1c630cf4-f669-4563-8483-f9f367841185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" Oct 30 05:15:00.373021 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.372983 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48947e23-be16-44b9-9c61-209679df23a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2\" (UID: \"48947e23-be16-44b9-9c61-209679df23a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" Oct 30 05:15:00.373021 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.373020 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzhwh\" (UniqueName: \"kubernetes.io/projected/48947e23-be16-44b9-9c61-209679df23a9-kube-api-access-gzhwh\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2\" (UID: \"48947e23-be16-44b9-9c61-209679df23a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" Oct 30 05:15:00.373258 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.373053 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvlqg\" (UniqueName: \"kubernetes.io/projected/4259a987-fbcc-4edf-867e-0f21fff10f9f-kube-api-access-rvlqg\") pod \"maas-api-key-cleanup-29888955-sp49s\" (UID: \"4259a987-fbcc-4edf-867e-0f21fff10f9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" Oct 30 05:15:00.373258 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.373070 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4259a987-fbcc-4edf-867e-0f21fff10f9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888955-sp49s\" (UID: \"4259a987-fbcc-4edf-867e-0f21fff10f9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" Oct 30 05:15:00.373258 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.373094 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbb2w\" (UniqueName: \"kubernetes.io/projected/1c630cf4-f669-4563-8483-f9f367841185-kube-api-access-dbb2w\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz\" (UID: \"1c630cf4-f669-4563-8483-f9f367841185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" Oct 30 05:15:00.373258 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.373133 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c630cf4-f669-4563-8483-f9f367841185-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz\" (UID: \"1c630cf4-f669-4563-8483-f9f367841185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" Oct 30 05:15:00.373787 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.373763 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4259a987-fbcc-4edf-867e-0f21fff10f9f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888955-sp49s\" (UID: \"4259a987-fbcc-4edf-867e-0f21fff10f9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" Oct 30 05:15:00.373787 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.373776 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c630cf4-f669-4563-8483-f9f367841185-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz\" (UID: \"1c630cf4-f669-4563-8483-f9f367841185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" Oct 30 05:15:00.373911 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.373771 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48947e23-be16-44b9-9c61-209679df23a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2\" (UID: \"48947e23-be16-44b9-9c61-209679df23a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" Oct 30 05:15:00.396230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.396203 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbb2w\" (UniqueName: \"kubernetes.io/projected/1c630cf4-f669-4563-8483-f9f367841185-kube-api-access-dbb2w\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz\" (UID: \"1c630cf4-f669-4563-8483-f9f367841185\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" Oct 30 05:15:00.396325 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.396250 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvlqg\" (UniqueName: \"kubernetes.io/projected/4259a987-fbcc-4edf-867e-0f21fff10f9f-kube-api-access-rvlqg\") pod \"maas-api-key-cleanup-29888955-sp49s\" (UID: \"4259a987-fbcc-4edf-867e-0f21fff10f9f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" Oct 30 05:15:00.396325 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.396299 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzhwh\" (UniqueName: \"kubernetes.io/projected/48947e23-be16-44b9-9c61-209679df23a9-kube-api-access-gzhwh\") pod \"maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2\" (UID: \"48947e23-be16-44b9-9c61-209679df23a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" Oct 30 05:15:00.460243 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.460214 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:00.481940 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.481876 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" Oct 30 05:15:00.500359 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.500335 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" Oct 30 05:15:00.502453 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.502429 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" Oct 30 05:15:00.674106 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.674078 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:00.789409 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.789208 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s"] Oct 30 05:15:00.791315 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:15:00.791283 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4259a987_fbcc_4edf_867e_0f21fff10f9f.slice/crio-982b7d9eb44842fea56e265eed824882b3852ef4d0a1105266bbde529ce1f3a1 WatchSource:0}: Error finding container 982b7d9eb44842fea56e265eed824882b3852ef4d0a1105266bbde529ce1f3a1: Status 404 returned error can't find the container with id 982b7d9eb44842fea56e265eed824882b3852ef4d0a1105266bbde529ce1f3a1 Oct 30 05:15:00.813521 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.813496 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz"] Oct 30 05:15:00.819512 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:15:00.819484 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c630cf4_f669_4563_8483_f9f367841185.slice/crio-91025bbd98510df8e217fdca0a77006309834325ee2552c9169f1b04a7ebc199 WatchSource:0}: Error finding container 91025bbd98510df8e217fdca0a77006309834325ee2552c9169f1b04a7ebc199: Status 404 returned error can't find the container with id 91025bbd98510df8e217fdca0a77006309834325ee2552c9169f1b04a7ebc199 Oct 30 05:15:00.825685 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.825664 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2"] Oct 30 05:15:00.830672 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:15:00.830644 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48947e23_be16_44b9_9c61_209679df23a9.slice/crio-0eec47630abaa197594f1bed6c2d6d998308f82cc4f39034d6c9623de544912a WatchSource:0}: Error finding container 0eec47630abaa197594f1bed6c2d6d998308f82cc4f39034d6c9623de544912a: Status 404 returned error can't find the container with id 0eec47630abaa197594f1bed6c2d6d998308f82cc4f39034d6c9623de544912a Oct 30 05:15:00.847543 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.847504 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc4dabd-cfc2-4a79-aadc-28833d9f99ef" path="/var/lib/kubelet/pods/cdc4dabd-cfc2-4a79-aadc-28833d9f99ef/volumes" Oct 30 05:15:00.899343 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.899308 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh"] Oct 30 05:15:00.899623 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.899590 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" containerID="cri-o://d8bf977300ac8be66c18c46874d345f263b403eaa09636344bf9f74809dbb559" gracePeriod=30 Oct 30 05:15:00.989461 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.989430 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" event={"ID":"48947e23-be16-44b9-9c61-209679df23a9","Type":"ContainerStarted","Data":"0eec47630abaa197594f1bed6c2d6d998308f82cc4f39034d6c9623de544912a"} Oct 30 05:15:00.990751 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.990726 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" event={"ID":"1c630cf4-f669-4563-8483-f9f367841185","Type":"ContainerStarted","Data":"91025bbd98510df8e217fdca0a77006309834325ee2552c9169f1b04a7ebc199"} Oct 30 05:15:00.991920 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:00.991898 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" event={"ID":"4259a987-fbcc-4edf-867e-0f21fff10f9f","Type":"ContainerStarted","Data":"982b7d9eb44842fea56e265eed824882b3852ef4d0a1105266bbde529ce1f3a1"} Oct 30 05:15:01.052766 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:01.052673 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:01.575867 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:01.575820 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:02.006493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:02.006457 4793 generic.go:356] "Generic (PLEG): container finished" podID="48947e23-be16-44b9-9c61-209679df23a9" containerID="57eab7eaba67c0afc11ee7a6621fb6d663aa53363cad5be0c5b04a488b424e58" exitCode=60 Oct 30 05:15:02.006986 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:02.006551 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" event={"ID":"48947e23-be16-44b9-9c61-209679df23a9","Type":"ContainerDied","Data":"57eab7eaba67c0afc11ee7a6621fb6d663aa53363cad5be0c5b04a488b424e58"} Oct 30 05:15:02.006986 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:02.006789 4793 scope.go:122] "RemoveContainer" containerID="57eab7eaba67c0afc11ee7a6621fb6d663aa53363cad5be0c5b04a488b424e58" Oct 30 05:15:02.010706 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:02.009731 4793 generic.go:356] "Generic (PLEG): container finished" podID="1c630cf4-f669-4563-8483-f9f367841185" containerID="7939722a4ad8b34eb58c03b756d2626fd78feb2d9ee6873ac531bbc2cfbf7954" exitCode=60 Oct 30 05:15:02.010706 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:02.010294 4793 scope.go:122] "RemoveContainer" containerID="7939722a4ad8b34eb58c03b756d2626fd78feb2d9ee6873ac531bbc2cfbf7954" Oct 30 05:15:02.010706 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:02.010631 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" event={"ID":"1c630cf4-f669-4563-8483-f9f367841185","Type":"ContainerDied","Data":"7939722a4ad8b34eb58c03b756d2626fd78feb2d9ee6873ac531bbc2cfbf7954"} Oct 30 05:15:02.014825 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:02.014798 4793 generic.go:356] "Generic (PLEG): container finished" podID="4259a987-fbcc-4edf-867e-0f21fff10f9f" containerID="688dc065a763eefdba544cd2e9b19877385a99618d3bbc2efd97a8ec0e47fe2a" exitCode=60 Oct 30 05:15:02.014957 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:02.014857 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" event={"ID":"4259a987-fbcc-4edf-867e-0f21fff10f9f","Type":"ContainerDied","Data":"688dc065a763eefdba544cd2e9b19877385a99618d3bbc2efd97a8ec0e47fe2a"} Oct 30 05:15:02.015118 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:02.015095 4793 scope.go:122] "RemoveContainer" containerID="688dc065a763eefdba544cd2e9b19877385a99618d3bbc2efd97a8ec0e47fe2a" Oct 30 05:15:02.104570 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:02.104536 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz"] Oct 30 05:15:02.181308 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:02.181273 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:02.838072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:02.838034 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:03.022831 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:03.022798 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" event={"ID":"4259a987-fbcc-4edf-867e-0f21fff10f9f","Type":"ContainerStarted","Data":"9351610c30aa777261173ecadc3eb62877e3cae32383cda42a8320f45209ce51"} Oct 30 05:15:03.024883 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:03.024859 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" event={"ID":"48947e23-be16-44b9-9c61-209679df23a9","Type":"ContainerStarted","Data":"89e92525212b2b039694c3b4c98660c3f5a850df64e5215ee01b4c7aa9c9f3f1"} Oct 30 05:15:03.026523 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:03.026498 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" event={"ID":"1c630cf4-f669-4563-8483-f9f367841185","Type":"ContainerStarted","Data":"3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2"} Oct 30 05:15:03.026643 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:03.026587 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" podUID="1c630cf4-f669-4563-8483-f9f367841185" containerName="cleanup" containerID="cri-o://3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2" gracePeriod=30 Oct 30 05:15:03.063800 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:03.063757 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" podStartSLOduration=3.063745968 podStartE2EDuration="3.063745968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.061401566 +0000 UTC m=+4626.347276338" watchObservedRunningTime="2026-10-30 05:15:03.063745968 +0000 UTC m=+4626.349620743" Oct 30 05:15:03.101790 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:03.101713 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" podStartSLOduration=3.101700074 podStartE2EDuration="3.101700074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.099768233 +0000 UTC m=+4626.385643008" watchObservedRunningTime="2026-10-30 05:15:03.101700074 +0000 UTC m=+4626.387574855" Oct 30 05:15:03.139503 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:03.139454 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" podStartSLOduration=3.139441719 podStartE2EDuration="3.139441719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:03.137894328 +0000 UTC m=+4626.423769104" watchObservedRunningTime="2026-10-30 05:15:03.139441719 +0000 UTC m=+4626.425316504" Oct 30 05:15:03.848306 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:03.848271 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:03.943108 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:03.943075 4793 patch_prober.go:29] interesting pod/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:15:03.943281 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:03.943133 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:15:04.034892 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:04.034860 4793 generic.go:356] "Generic (PLEG): container finished" podID="4259a987-fbcc-4edf-867e-0f21fff10f9f" containerID="9351610c30aa777261173ecadc3eb62877e3cae32383cda42a8320f45209ce51" exitCode=60 Oct 30 05:15:04.035359 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:04.034914 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" event={"ID":"4259a987-fbcc-4edf-867e-0f21fff10f9f","Type":"ContainerDied","Data":"9351610c30aa777261173ecadc3eb62877e3cae32383cda42a8320f45209ce51"} Oct 30 05:15:04.035359 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:04.034978 4793 scope.go:122] "RemoveContainer" containerID="688dc065a763eefdba544cd2e9b19877385a99618d3bbc2efd97a8ec0e47fe2a" Oct 30 05:15:04.035359 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:04.035289 4793 scope.go:122] "RemoveContainer" containerID="9351610c30aa777261173ecadc3eb62877e3cae32383cda42a8320f45209ce51" Oct 30 05:15:04.035621 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:04.035599 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888955-sp49s_odh-ai-gateway-infra(4259a987-fbcc-4edf-867e-0f21fff10f9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" Oct 30 05:15:04.037246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:04.037222 4793 generic.go:356] "Generic (PLEG): container finished" podID="48947e23-be16-44b9-9c61-209679df23a9" containerID="89e92525212b2b039694c3b4c98660c3f5a850df64e5215ee01b4c7aa9c9f3f1" exitCode=60 Oct 30 05:15:04.037362 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:04.037265 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" event={"ID":"48947e23-be16-44b9-9c61-209679df23a9","Type":"ContainerDied","Data":"89e92525212b2b039694c3b4c98660c3f5a850df64e5215ee01b4c7aa9c9f3f1"} Oct 30 05:15:04.037582 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:04.037566 4793 scope.go:122] "RemoveContainer" containerID="89e92525212b2b039694c3b4c98660c3f5a850df64e5215ee01b4c7aa9c9f3f1" Oct 30 05:15:04.037822 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:04.037802 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2_odh-ai-gateway-infra(48947e23-be16-44b9-9c61-209679df23a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" podUID="48947e23-be16-44b9-9c61-209679df23a9" Oct 30 05:15:04.048546 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:04.048529 4793 scope.go:122] "RemoveContainer" containerID="57eab7eaba67c0afc11ee7a6621fb6d663aa53363cad5be0c5b04a488b424e58" Oct 30 05:15:04.099307 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:04.099231 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:04.571274 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:04.571218 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:04.782057 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:04.782020 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:05.000476 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.000428 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:05.044777 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.044755 4793 scope.go:122] "RemoveContainer" containerID="89e92525212b2b039694c3b4c98660c3f5a850df64e5215ee01b4c7aa9c9f3f1" Oct 30 05:15:05.045165 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:05.045016 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2_odh-ai-gateway-infra(48947e23-be16-44b9-9c61-209679df23a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" podUID="48947e23-be16-44b9-9c61-209679df23a9" Oct 30 05:15:05.046339 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.046321 4793 scope.go:122] "RemoveContainer" containerID="9351610c30aa777261173ecadc3eb62877e3cae32383cda42a8320f45209ce51" Oct 30 05:15:05.046542 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:05.046523 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888955-sp49s_odh-ai-gateway-infra(4259a987-fbcc-4edf-867e-0f21fff10f9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" Oct 30 05:15:05.407358 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.407265 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:05.732466 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.732431 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr"] Oct 30 05:15:05.737851 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.737830 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:05.752574 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.752548 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:15:05.753714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.753691 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr"] Oct 30 05:15:05.833364 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.833323 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4czj\" (UniqueName: \"kubernetes.io/projected/29fd98c6-e26f-4d08-a688-0f3b31404ebd-kube-api-access-x4czj\") pod \"maas-api-revoke-sub-8dea38ec-5lpvr\" (UID: \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:05.833563 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.833470 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dea38ec-5lpvr\" (UID: \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:05.833651 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.833634 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dea38ec-5lpvr\" (UID: \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:05.934198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.934165 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dea38ec-5lpvr\" (UID: \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:05.934338 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.934211 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4czj\" (UniqueName: \"kubernetes.io/projected/29fd98c6-e26f-4d08-a688-0f3b31404ebd-kube-api-access-x4czj\") pod \"maas-api-revoke-sub-8dea38ec-5lpvr\" (UID: \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:05.934338 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.934293 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dea38ec-5lpvr\" (UID: \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:05.934950 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.934926 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8dea38ec-5lpvr\" (UID: \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:05.936448 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.936428 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8dea38ec-5lpvr\" (UID: \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:05.956785 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:05.956762 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4czj\" (UniqueName: \"kubernetes.io/projected/29fd98c6-e26f-4d08-a688-0f3b31404ebd-kube-api-access-x4czj\") pod \"maas-api-revoke-sub-8dea38ec-5lpvr\" (UID: \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:06.054712 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.054680 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:06.055196 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.054742 4793 generic.go:356] "Generic (PLEG): container finished" podID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerID="d8bf977300ac8be66c18c46874d345f263b403eaa09636344bf9f74809dbb559" exitCode=0 Oct 30 05:15:06.055196 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.054830 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" event={"ID":"ae86f996-7f99-491c-b774-8b4b97dea5ee","Type":"ContainerDied","Data":"d8bf977300ac8be66c18c46874d345f263b403eaa09636344bf9f74809dbb559"} Oct 30 05:15:06.055196 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.054910 4793 scope.go:122] "RemoveContainer" containerID="f3f8f92ed658fb381b227ac0304def009e7d48d25d5e711d6c8dff6e6ee7d40f" Oct 30 05:15:06.175846 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.175820 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 05:15:06.236783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.236756 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-envoy\") pod \"ae86f996-7f99-491c-b774-8b4b97dea5ee\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " Oct 30 05:15:06.236928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.236862 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-certs\") pod \"ae86f996-7f99-491c-b774-8b4b97dea5ee\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " Oct 30 05:15:06.236928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.236899 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-data\") pod \"ae86f996-7f99-491c-b774-8b4b97dea5ee\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " Oct 30 05:15:06.237058 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.236935 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-podinfo\") pod \"ae86f996-7f99-491c-b774-8b4b97dea5ee\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " Oct 30 05:15:06.237058 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.236974 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-socket\") pod \"ae86f996-7f99-491c-b774-8b4b97dea5ee\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " Oct 30 05:15:06.237058 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.237024 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-credential-socket\") pod \"ae86f996-7f99-491c-b774-8b4b97dea5ee\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " Oct 30 05:15:06.237232 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.237080 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-token\" (UniqueName: \"kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-token\") pod \"ae86f996-7f99-491c-b774-8b4b97dea5ee\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " Oct 30 05:15:06.237232 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.237130 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-kube-api-access-h9hrf\" (UniqueName: \"kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-kube-api-access-h9hrf\") pod \"ae86f996-7f99-491c-b774-8b4b97dea5ee\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " Oct 30 05:15:06.237232 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.237155 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-certs" pod "ae86f996-7f99-491c-b774-8b4b97dea5ee" (UID: "ae86f996-7f99-491c-b774-8b4b97dea5ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:06.237232 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.237163 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae86f996-7f99-491c-b774-8b4b97dea5ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae86f996-7f99-491c-b774-8b4b97dea5ee-istiod-ca-cert\") pod \"ae86f996-7f99-491c-b774-8b4b97dea5ee\" (UID: \"ae86f996-7f99-491c-b774-8b4b97dea5ee\") " Oct 30 05:15:06.237570 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.237529 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae86f996-7f99-491c-b774-8b4b97dea5ee-istiod-ca-cert" pod "ae86f996-7f99-491c-b774-8b4b97dea5ee" (UID: "ae86f996-7f99-491c-b774-8b4b97dea5ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:06.237640 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.237588 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-socket" pod "ae86f996-7f99-491c-b774-8b4b97dea5ee" (UID: "ae86f996-7f99-491c-b774-8b4b97dea5ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:06.237728 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.237198 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-data" pod "ae86f996-7f99-491c-b774-8b4b97dea5ee" (UID: "ae86f996-7f99-491c-b774-8b4b97dea5ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:06.237780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.237715 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:06.237780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.237749 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:06.237780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.237761 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae86f996-7f99-491c-b774-8b4b97dea5ee-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:06.237780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.237730 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-credential-socket" pod "ae86f996-7f99-491c-b774-8b4b97dea5ee" (UID: "ae86f996-7f99-491c-b774-8b4b97dea5ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:06.239196 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.239165 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-envoy" pod "ae86f996-7f99-491c-b774-8b4b97dea5ee" (UID: "ae86f996-7f99-491c-b774-8b4b97dea5ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:06.239670 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.239652 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-podinfo" pod "ae86f996-7f99-491c-b774-8b4b97dea5ee" (UID: "ae86f996-7f99-491c-b774-8b4b97dea5ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:15:06.239979 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.239955 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-kube-api-access-h9hrf" pod "ae86f996-7f99-491c-b774-8b4b97dea5ee" (UID: "ae86f996-7f99-491c-b774-8b4b97dea5ee"). InnerVolumeSpecName "kube-api-access-h9hrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:06.240073 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.240012 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-token" pod "ae86f996-7f99-491c-b774-8b4b97dea5ee" (UID: "ae86f996-7f99-491c-b774-8b4b97dea5ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:06.339020 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.338953 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:06.339020 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.338982 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:06.339020 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.338992 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:06.339020 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.339002 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:06.339020 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.339011 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9hrf\" (UniqueName: \"kubernetes.io/projected/ae86f996-7f99-491c-b774-8b4b97dea5ee-kube-api-access-h9hrf\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:06.339020 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.339020 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae86f996-7f99-491c-b774-8b4b97dea5ee-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:06.377664 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.377632 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr"] Oct 30 05:15:06.381493 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:15:06.381463 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fd98c6_e26f_4d08_a688_0f3b31404ebd.slice/crio-ff80c98903fcac588be7c7836608f8da65629cf2225d3220a4b68bd6e3c18ede WatchSource:0}: Error finding container ff80c98903fcac588be7c7836608f8da65629cf2225d3220a4b68bd6e3c18ede: Status 404 returned error can't find the container with id ff80c98903fcac588be7c7836608f8da65629cf2225d3220a4b68bd6e3c18ede Oct 30 05:15:06.411784 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.411761 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs"] Oct 30 05:15:06.515301 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.515272 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:06.809509 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:06.809476 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:07.026409 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:07.026374 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:07.064728 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:07.064630 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" event={"ID":"29fd98c6-e26f-4d08-a688-0f3b31404ebd","Type":"ContainerStarted","Data":"43323a2877bf62f0ebf43a22f82d4536c7220e9f6f5ec379f9925a3caca09745"} Oct 30 05:15:07.064728 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:07.064698 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" event={"ID":"29fd98c6-e26f-4d08-a688-0f3b31404ebd","Type":"ContainerStarted","Data":"ff80c98903fcac588be7c7836608f8da65629cf2225d3220a4b68bd6e3c18ede"} Oct 30 05:15:07.066320 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:07.066291 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" event={"ID":"ae86f996-7f99-491c-b774-8b4b97dea5ee","Type":"ContainerDied","Data":"4faa656bb0a633abcb934dc91ff7b62be5ab28ac687e97cd243ad7d5a8a377c2"} Oct 30 05:15:07.066492 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:07.066321 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh" Oct 30 05:15:07.066492 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:07.066352 4793 scope.go:122] "RemoveContainer" containerID="d8bf977300ac8be66c18c46874d345f263b403eaa09636344bf9f74809dbb559" Oct 30 05:15:07.104923 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:07.104877 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" podStartSLOduration=2.104862689 podStartE2EDuration="2.104862689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:07.102738823 +0000 UTC m=+4630.388613625" watchObservedRunningTime="2026-10-30 05:15:07.104862689 +0000 UTC m=+4630.390737463" Oct 30 05:15:07.138973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:07.138947 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh"] Oct 30 05:15:07.150325 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:07.150300 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-247158-openshift-default-7d589d7779-ljdgh"] Oct 30 05:15:07.341788 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:07.341698 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:08.075984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:08.075952 4793 generic.go:356] "Generic (PLEG): container finished" podID="29fd98c6-e26f-4d08-a688-0f3b31404ebd" containerID="43323a2877bf62f0ebf43a22f82d4536c7220e9f6f5ec379f9925a3caca09745" exitCode=0 Oct 30 05:15:08.076487 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:08.076029 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" event={"ID":"29fd98c6-e26f-4d08-a688-0f3b31404ebd","Type":"ContainerDied","Data":"43323a2877bf62f0ebf43a22f82d4536c7220e9f6f5ec379f9925a3caca09745"} Oct 30 05:15:08.356288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:08.356195 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:08.845097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:08.845061 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" path="/var/lib/kubelet/pods/ae86f996-7f99-491c-b774-8b4b97dea5ee/volumes" Oct 30 05:15:09.211809 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:09.211785 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:09.366812 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:09.366777 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-openshift-ca\") pod \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\" (UID: \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\") " Oct 30 05:15:09.367004 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:09.366840 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-xks-ca\") pod \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\" (UID: \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\") " Oct 30 05:15:09.367004 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:09.366918 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29fd98c6-e26f-4d08-a688-0f3b31404ebd-kube-api-access-x4czj\" (UniqueName: \"kubernetes.io/projected/29fd98c6-e26f-4d08-a688-0f3b31404ebd-kube-api-access-x4czj\") pod \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\" (UID: \"29fd98c6-e26f-4d08-a688-0f3b31404ebd\") " Oct 30 05:15:09.367126 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:09.367103 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-openshift-ca" pod "29fd98c6-e26f-4d08-a688-0f3b31404ebd" (UID: "29fd98c6-e26f-4d08-a688-0f3b31404ebd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:09.367304 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:09.367286 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:09.369013 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:09.368981 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fd98c6-e26f-4d08-a688-0f3b31404ebd-kube-api-access-x4czj" pod "29fd98c6-e26f-4d08-a688-0f3b31404ebd" (UID: "29fd98c6-e26f-4d08-a688-0f3b31404ebd"). InnerVolumeSpecName "kube-api-access-x4czj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:09.369127 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:09.369043 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-xks-ca" pod "29fd98c6-e26f-4d08-a688-0f3b31404ebd" (UID: "29fd98c6-e26f-4d08-a688-0f3b31404ebd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:15:09.468595 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:09.468556 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4czj\" (UniqueName: \"kubernetes.io/projected/29fd98c6-e26f-4d08-a688-0f3b31404ebd-kube-api-access-x4czj\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:09.468731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:09.468621 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29fd98c6-e26f-4d08-a688-0f3b31404ebd-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:09.917452 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:09.917325 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:10.092729 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:10.092697 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" Oct 30 05:15:10.092729 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:10.092716 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr" event={"ID":"29fd98c6-e26f-4d08-a688-0f3b31404ebd","Type":"ContainerDied","Data":"ff80c98903fcac588be7c7836608f8da65629cf2225d3220a4b68bd6e3c18ede"} Oct 30 05:15:10.092981 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:10.092778 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff80c98903fcac588be7c7836608f8da65629cf2225d3220a4b68bd6e3c18ede" Oct 30 05:15:10.259854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:10.259818 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f"] Oct 30 05:15:10.268907 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:10.268876 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz"] Oct 30 05:15:10.496206 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:10.496175 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:11.072336 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:11.072300 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:11.218923 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:11.218890 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:15:11.613214 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:11.613168 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:13.308456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.308411 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:13.789777 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.789741 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg"] Oct 30 05:15:13.790445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.790406 4793 cpu_manager.go:414] "containerMap: removing container" podUID="29fd98c6-e26f-4d08-a688-0f3b31404ebd" containerName="revoke-keys" Oct 30 05:15:13.790445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.790441 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fd98c6-e26f-4d08-a688-0f3b31404ebd" containerName="revoke-keys" Oct 30 05:15:13.790606 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.790459 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" Oct 30 05:15:13.790606 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.790465 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" Oct 30 05:15:13.790606 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.790541 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fd98c6-e26f-4d08-a688-0f3b31404ebd" containerName="revoke-keys" Oct 30 05:15:13.790606 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.790550 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" Oct 30 05:15:13.790606 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.790560 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" Oct 30 05:15:13.793588 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.793569 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:13.808844 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.808819 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:15:13.824680 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.824654 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg"] Oct 30 05:15:13.838888 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.838863 4793 scope.go:122] "RemoveContainer" containerID="9351610c30aa777261173ecadc3eb62877e3cae32383cda42a8320f45209ce51" Oct 30 05:15:13.839036 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.838909 4793 scope.go:122] "RemoveContainer" containerID="89e92525212b2b039694c3b4c98660c3f5a850df64e5215ee01b4c7aa9c9f3f1" Oct 30 05:15:13.914319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.913984 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgm24\" (UniqueName: \"kubernetes.io/projected/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-kube-api-access-mgm24\") pod \"maas-api-revoke-sub-cb533423-chhrg\" (UID: \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:13.914319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.914095 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb533423-chhrg\" (UID: \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:13.914319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:13.914138 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb533423-chhrg\" (UID: \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:14.015833 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:14.015794 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgm24\" (UniqueName: \"kubernetes.io/projected/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-kube-api-access-mgm24\") pod \"maas-api-revoke-sub-cb533423-chhrg\" (UID: \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:14.016016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:14.015925 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb533423-chhrg\" (UID: \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:14.016016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:14.015972 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb533423-chhrg\" (UID: \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:14.018203 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:14.018165 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb533423-chhrg\" (UID: \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:14.019601 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:14.019542 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb533423-chhrg\" (UID: \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:14.039199 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:14.039167 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgm24\" (UniqueName: \"kubernetes.io/projected/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-kube-api-access-mgm24\") pod \"maas-api-revoke-sub-cb533423-chhrg\" (UID: \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:14.110027 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:14.109999 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:14.452566 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:14.452540 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg"] Oct 30 05:15:14.977317 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:14.977283 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:15.048214 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.048134 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg"] Oct 30 05:15:15.071195 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.071167 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q"] Oct 30 05:15:15.128892 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.128650 4793 generic.go:356] "Generic (PLEG): container finished" podID="48947e23-be16-44b9-9c61-209679df23a9" containerID="343485c5a8fd9bc4b270003d7930a9150fde8c0a1ac54deda11fc0fe46e25e7b" exitCode=60 Oct 30 05:15:15.128892 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.128857 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" event={"ID":"48947e23-be16-44b9-9c61-209679df23a9","Type":"ContainerDied","Data":"343485c5a8fd9bc4b270003d7930a9150fde8c0a1ac54deda11fc0fe46e25e7b"} Oct 30 05:15:15.129191 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.128914 4793 scope.go:122] "RemoveContainer" containerID="89e92525212b2b039694c3b4c98660c3f5a850df64e5215ee01b4c7aa9c9f3f1" Oct 30 05:15:15.129387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.129367 4793 scope.go:122] "RemoveContainer" containerID="343485c5a8fd9bc4b270003d7930a9150fde8c0a1ac54deda11fc0fe46e25e7b" Oct 30 05:15:15.129685 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:15.129659 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2_odh-ai-gateway-infra(48947e23-be16-44b9-9c61-209679df23a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" podUID="48947e23-be16-44b9-9c61-209679df23a9" Oct 30 05:15:15.133616 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.133593 4793 generic.go:356] "Generic (PLEG): container finished" podID="4259a987-fbcc-4edf-867e-0f21fff10f9f" containerID="51dfac5021f2cc6943255ef113a34126f309458e07f8752cdeadc4a88902d64e" exitCode=60 Oct 30 05:15:15.133723 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.133676 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" event={"ID":"4259a987-fbcc-4edf-867e-0f21fff10f9f","Type":"ContainerDied","Data":"51dfac5021f2cc6943255ef113a34126f309458e07f8752cdeadc4a88902d64e"} Oct 30 05:15:15.134045 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.134018 4793 scope.go:122] "RemoveContainer" containerID="51dfac5021f2cc6943255ef113a34126f309458e07f8752cdeadc4a88902d64e" Oct 30 05:15:15.134236 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:15.134216 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888955-sp49s_odh-ai-gateway-infra(4259a987-fbcc-4edf-867e-0f21fff10f9f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" Oct 30 05:15:15.135802 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.135777 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" event={"ID":"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79","Type":"ContainerStarted","Data":"89bdde8540de40ec9af8f3d31e9aad441fe4fdbdb4a1b28efa7df97aec4b759e"} Oct 30 05:15:15.135919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.135824 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" event={"ID":"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79","Type":"ContainerStarted","Data":"3a26958df6cdbce099978124858b3eefa3bd4e5e3a7ca83046080305b25d130a"} Oct 30 05:15:15.146436 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.146397 4793 scope.go:122] "RemoveContainer" containerID="9351610c30aa777261173ecadc3eb62877e3cae32383cda42a8320f45209ce51" Oct 30 05:15:15.235133 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.235101 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:15.445458 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:15.445404 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:16.144461 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.144411 4793 generic.go:356] "Generic (PLEG): container finished" podID="99b2e7fe-1ed9-471e-8169-3ce58d8c7c79" containerID="89bdde8540de40ec9af8f3d31e9aad441fe4fdbdb4a1b28efa7df97aec4b759e" exitCode=0 Oct 30 05:15:16.144938 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.144459 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" event={"ID":"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79","Type":"ContainerDied","Data":"89bdde8540de40ec9af8f3d31e9aad441fe4fdbdb4a1b28efa7df97aec4b759e"} Oct 30 05:15:16.180517 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.180472 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" podStartSLOduration=3.180454221 podStartE2EDuration="3.180454221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:15.247374152 +0000 UTC m=+4638.533248996" watchObservedRunningTime="2026-10-30 05:15:16.180454221 +0000 UTC m=+4639.466328996" Oct 30 05:15:16.180695 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.180681 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2"] Oct 30 05:15:16.218844 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.218812 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s"] Oct 30 05:15:16.336998 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.336971 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" Oct 30 05:15:16.387539 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.387513 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" Oct 30 05:15:16.439722 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.439694 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48947e23-be16-44b9-9c61-209679df23a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48947e23-be16-44b9-9c61-209679df23a9-maas-api-service-ca\") pod \"48947e23-be16-44b9-9c61-209679df23a9\" (UID: \"48947e23-be16-44b9-9c61-209679df23a9\") " Oct 30 05:15:16.439858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.439738 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48947e23-be16-44b9-9c61-209679df23a9-kube-api-access-gzhwh\" (UniqueName: \"kubernetes.io/projected/48947e23-be16-44b9-9c61-209679df23a9-kube-api-access-gzhwh\") pod \"48947e23-be16-44b9-9c61-209679df23a9\" (UID: \"48947e23-be16-44b9-9c61-209679df23a9\") " Oct 30 05:15:16.440152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.440124 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48947e23-be16-44b9-9c61-209679df23a9-maas-api-service-ca" pod "48947e23-be16-44b9-9c61-209679df23a9" (UID: "48947e23-be16-44b9-9c61-209679df23a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:16.441817 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.441791 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48947e23-be16-44b9-9c61-209679df23a9-kube-api-access-gzhwh" pod "48947e23-be16-44b9-9c61-209679df23a9" (UID: "48947e23-be16-44b9-9c61-209679df23a9"). InnerVolumeSpecName "kube-api-access-gzhwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:16.540573 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.540546 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4259a987-fbcc-4edf-867e-0f21fff10f9f-kube-api-access-rvlqg\" (UniqueName: \"kubernetes.io/projected/4259a987-fbcc-4edf-867e-0f21fff10f9f-kube-api-access-rvlqg\") pod \"4259a987-fbcc-4edf-867e-0f21fff10f9f\" (UID: \"4259a987-fbcc-4edf-867e-0f21fff10f9f\") " Oct 30 05:15:16.540718 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.540707 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4259a987-fbcc-4edf-867e-0f21fff10f9f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4259a987-fbcc-4edf-867e-0f21fff10f9f-maas-api-service-ca\") pod \"4259a987-fbcc-4edf-867e-0f21fff10f9f\" (UID: \"4259a987-fbcc-4edf-867e-0f21fff10f9f\") " Oct 30 05:15:16.540912 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.540897 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48947e23-be16-44b9-9c61-209679df23a9-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:16.540912 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.540911 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzhwh\" (UniqueName: \"kubernetes.io/projected/48947e23-be16-44b9-9c61-209679df23a9-kube-api-access-gzhwh\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:16.541045 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.541024 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4259a987-fbcc-4edf-867e-0f21fff10f9f-maas-api-service-ca" pod "4259a987-fbcc-4edf-867e-0f21fff10f9f" (UID: "4259a987-fbcc-4edf-867e-0f21fff10f9f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:16.542557 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.542537 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4259a987-fbcc-4edf-867e-0f21fff10f9f-kube-api-access-rvlqg" pod "4259a987-fbcc-4edf-867e-0f21fff10f9f" (UID: "4259a987-fbcc-4edf-867e-0f21fff10f9f"). InnerVolumeSpecName "kube-api-access-rvlqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:16.642355 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.642330 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4259a987-fbcc-4edf-867e-0f21fff10f9f-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:16.642355 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:16.642354 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvlqg\" (UniqueName: \"kubernetes.io/projected/4259a987-fbcc-4edf-867e-0f21fff10f9f-kube-api-access-rvlqg\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:17.010402 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.010358 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:17.153733 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.153643 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" Oct 30 05:15:17.153733 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.153662 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s" event={"ID":"4259a987-fbcc-4edf-867e-0f21fff10f9f","Type":"ContainerDied","Data":"982b7d9eb44842fea56e265eed824882b3852ef4d0a1105266bbde529ce1f3a1"} Oct 30 05:15:17.154231 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.153739 4793 scope.go:122] "RemoveContainer" containerID="51dfac5021f2cc6943255ef113a34126f309458e07f8752cdeadc4a88902d64e" Oct 30 05:15:17.155852 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.155624 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" Oct 30 05:15:17.155852 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.155688 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2" event={"ID":"48947e23-be16-44b9-9c61-209679df23a9","Type":"ContainerDied","Data":"0eec47630abaa197594f1bed6c2d6d998308f82cc4f39034d6c9623de544912a"} Oct 30 05:15:17.165692 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.165650 4793 scope.go:122] "RemoveContainer" containerID="343485c5a8fd9bc4b270003d7930a9150fde8c0a1ac54deda11fc0fe46e25e7b" Oct 30 05:15:17.211844 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.211814 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s"] Oct 30 05:15:17.223836 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.223810 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888955-sp49s"] Oct 30 05:15:17.256710 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.256682 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2"] Oct 30 05:15:17.260131 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.260102 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-b4708a-29888955-qqhk2"] Oct 30 05:15:17.287405 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.287371 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:17.434856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.434824 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h"] Oct 30 05:15:17.435452 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435435 4793 cpu_manager.go:414] "containerMap: removing container" podUID="48947e23-be16-44b9-9c61-209679df23a9" containerName="cleanup" Oct 30 05:15:17.435524 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435455 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48947e23-be16-44b9-9c61-209679df23a9" containerName="cleanup" Oct 30 05:15:17.435524 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435475 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" Oct 30 05:15:17.435524 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435482 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae86f996-7f99-491c-b774-8b4b97dea5ee" containerName="istio-proxy" Oct 30 05:15:17.435524 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435494 4793 cpu_manager.go:414] "containerMap: removing container" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" containerName="cleanup" Oct 30 05:15:17.435524 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435501 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" containerName="cleanup" Oct 30 05:15:17.435524 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435509 4793 cpu_manager.go:414] "containerMap: removing container" podUID="48947e23-be16-44b9-9c61-209679df23a9" containerName="cleanup" Oct 30 05:15:17.435524 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435517 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48947e23-be16-44b9-9c61-209679df23a9" containerName="cleanup" Oct 30 05:15:17.435734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435527 4793 cpu_manager.go:414] "containerMap: removing container" podUID="48947e23-be16-44b9-9c61-209679df23a9" containerName="cleanup" Oct 30 05:15:17.435734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435535 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48947e23-be16-44b9-9c61-209679df23a9" containerName="cleanup" Oct 30 05:15:17.435734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435560 4793 cpu_manager.go:414] "containerMap: removing container" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" containerName="cleanup" Oct 30 05:15:17.435734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435567 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" containerName="cleanup" Oct 30 05:15:17.435734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435587 4793 cpu_manager.go:414] "containerMap: removing container" podUID="99b2e7fe-1ed9-471e-8169-3ce58d8c7c79" containerName="revoke-keys" Oct 30 05:15:17.435734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435594 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b2e7fe-1ed9-471e-8169-3ce58d8c7c79" containerName="revoke-keys" Oct 30 05:15:17.435734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435685 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="48947e23-be16-44b9-9c61-209679df23a9" containerName="cleanup" Oct 30 05:15:17.435734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435695 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" containerName="cleanup" Oct 30 05:15:17.435734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435705 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="48947e23-be16-44b9-9c61-209679df23a9" containerName="cleanup" Oct 30 05:15:17.435734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435718 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" containerName="cleanup" Oct 30 05:15:17.435734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.435728 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b2e7fe-1ed9-471e-8169-3ce58d8c7c79" containerName="revoke-keys" Oct 30 05:15:17.439128 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.439108 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:17.456844 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.456818 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h"] Oct 30 05:15:17.458613 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.458585 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-xks-ca\") pod \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\" (UID: \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\") " Oct 30 05:15:17.458780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.458683 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-kube-api-access-mgm24\" (UniqueName: \"kubernetes.io/projected/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-kube-api-access-mgm24\") pod \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\" (UID: \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\") " Oct 30 05:15:17.458780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.458723 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-openshift-ca\") pod \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\" (UID: \"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79\") " Oct 30 05:15:17.460544 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.459643 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-openshift-ca" pod "99b2e7fe-1ed9-471e-8169-3ce58d8c7c79" (UID: "99b2e7fe-1ed9-471e-8169-3ce58d8c7c79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:17.461862 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.461836 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-kube-api-access-mgm24" pod "99b2e7fe-1ed9-471e-8169-3ce58d8c7c79" (UID: "99b2e7fe-1ed9-471e-8169-3ce58d8c7c79"). InnerVolumeSpecName "kube-api-access-mgm24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:17.465629 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.465607 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-xks-ca" pod "99b2e7fe-1ed9-471e-8169-3ce58d8c7c79" (UID: "99b2e7fe-1ed9-471e-8169-3ce58d8c7c79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:15:17.522834 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.522805 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5"] Oct 30 05:15:17.561126 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.561095 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff3af461-8s58h\" (UID: \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:17.561271 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.561186 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff3af461-8s58h\" (UID: \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:17.561329 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.561274 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsj69\" (UniqueName: \"kubernetes.io/projected/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-kube-api-access-zsj69\") pod \"maas-api-revoke-sub-ff3af461-8s58h\" (UID: \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:17.561410 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.561396 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:17.561471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.561431 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgm24\" (UniqueName: \"kubernetes.io/projected/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-kube-api-access-mgm24\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:17.561471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.561448 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:17.662254 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.662216 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff3af461-8s58h\" (UID: \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:17.662483 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.662267 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff3af461-8s58h\" (UID: \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:17.662483 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.662318 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsj69\" (UniqueName: \"kubernetes.io/projected/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-kube-api-access-zsj69\") pod \"maas-api-revoke-sub-ff3af461-8s58h\" (UID: \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:17.663069 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.663040 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff3af461-8s58h\" (UID: \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:17.664813 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.664779 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff3af461-8s58h\" (UID: \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:17.685463 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.685376 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsj69\" (UniqueName: \"kubernetes.io/projected/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-kube-api-access-zsj69\") pod \"maas-api-revoke-sub-ff3af461-8s58h\" (UID: \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:17.739943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.739906 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:17.752402 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.752377 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:17.961332 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:17.961242 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:18.055897 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:18.055864 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h"] Oct 30 05:15:18.057302 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:15:18.057273 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89a3b1a_e196_4cba_b7b3_3d0f248d2eb3.slice/crio-7173175d561e0327a4ec3c53bb9530f2683c3d3f2cd916cb509c0a394a7cf6df WatchSource:0}: Error finding container 7173175d561e0327a4ec3c53bb9530f2683c3d3f2cd916cb509c0a394a7cf6df: Status 404 returned error can't find the container with id 7173175d561e0327a4ec3c53bb9530f2683c3d3f2cd916cb509c0a394a7cf6df Oct 30 05:15:18.164585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:18.164546 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" event={"ID":"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3","Type":"ContainerStarted","Data":"7173175d561e0327a4ec3c53bb9530f2683c3d3f2cd916cb509c0a394a7cf6df"} Oct 30 05:15:18.167263 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:18.167239 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" Oct 30 05:15:18.167764 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:18.167734 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg" event={"ID":"99b2e7fe-1ed9-471e-8169-3ce58d8c7c79","Type":"ContainerDied","Data":"3a26958df6cdbce099978124858b3eefa3bd4e5e3a7ca83046080305b25d130a"} Oct 30 05:15:18.167860 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:18.167782 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a26958df6cdbce099978124858b3eefa3bd4e5e3a7ca83046080305b25d130a" Oct 30 05:15:18.845247 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:18.845213 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" path="/var/lib/kubelet/pods/4259a987-fbcc-4edf-867e-0f21fff10f9f/volumes" Oct 30 05:15:18.845666 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:18.845651 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48947e23-be16-44b9-9c61-209679df23a9" path="/var/lib/kubelet/pods/48947e23-be16-44b9-9c61-209679df23a9/volumes" Oct 30 05:15:19.175956 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:19.175924 4793 generic.go:356] "Generic (PLEG): container finished" podID="d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3" containerID="a19623f5f3697bb7a8e515adbdaa57c1a2fe1578d331670abe0801aa1286969a" exitCode=0 Oct 30 05:15:19.176328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:19.175982 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" event={"ID":"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3","Type":"ContainerDied","Data":"a19623f5f3697bb7a8e515adbdaa57c1a2fe1578d331670abe0801aa1286969a"} Oct 30 05:15:20.049462 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:20.049404 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" podUID="f186039e-f254-4792-bea7-3607de3d4a7f" containerName="payload-processing" containerID="cri-o://f989dc363e40c8c9888871988145afe8f7972bee92b8e0c080799992e007fa53" gracePeriod=25 Oct 30 05:15:20.071905 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:20.071869 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" podUID="73c14d0b-ca18-4835-94c4-aedc0c1ad8e7" containerName="payload-pre-processing" containerID="cri-o://ae3dee5aa118ee3dbcc4ccc2a67d9858a0d7825ed1482705d799877d15cfe1f3" gracePeriod=25 Oct 30 05:15:20.336213 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:20.336187 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:20.383226 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:20.383194 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-xks-ca\") pod \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\" (UID: \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\") " Oct 30 05:15:20.383392 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:20.383243 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-kube-api-access-zsj69\" (UniqueName: \"kubernetes.io/projected/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-kube-api-access-zsj69\") pod \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\" (UID: \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\") " Oct 30 05:15:20.383392 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:20.383318 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-openshift-ca\") pod \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\" (UID: \"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3\") " Oct 30 05:15:20.383963 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:20.383938 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-openshift-ca" pod "d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3" (UID: "d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:20.385599 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:20.385574 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-kube-api-access-zsj69" pod "d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3" (UID: "d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3"). InnerVolumeSpecName "kube-api-access-zsj69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:20.385847 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:20.385820 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-xks-ca" pod "d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3" (UID: "d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:15:20.484736 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:20.484700 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:20.484736 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:20.484730 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsj69\" (UniqueName: \"kubernetes.io/projected/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-kube-api-access-zsj69\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:20.484891 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:20.484744 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:21.059809 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.059776 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:21.189109 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.189079 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" Oct 30 05:15:21.189109 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.189088 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h" event={"ID":"d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3","Type":"ContainerDied","Data":"7173175d561e0327a4ec3c53bb9530f2683c3d3f2cd916cb509c0a394a7cf6df"} Oct 30 05:15:21.189355 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.189138 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7173175d561e0327a4ec3c53bb9530f2683c3d3f2cd916cb509c0a394a7cf6df" Oct 30 05:15:21.431049 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.431000 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" podUID="439ce07a-ea3d-4b97-a8ed-7655a840c873" containerName="main" containerID="cri-o://0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12" gracePeriod=45 Oct 30 05:15:21.689013 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.688988 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 05:15:21.695146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.695100 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/439ce07a-ea3d-4b97-a8ed-7655a840c873-tls-certs\" (UniqueName: \"kubernetes.io/secret/439ce07a-ea3d-4b97-a8ed-7655a840c873-tls-certs\") pod \"439ce07a-ea3d-4b97-a8ed-7655a840c873\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " Oct 30 05:15:21.695146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.695134 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-model-cache\") pod \"439ce07a-ea3d-4b97-a8ed-7655a840c873\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " Oct 30 05:15:21.695312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.695165 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-dshm\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-dshm\") pod \"439ce07a-ea3d-4b97-a8ed-7655a840c873\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " Oct 30 05:15:21.695312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.695188 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/439ce07a-ea3d-4b97-a8ed-7655a840c873-kube-api-access-ndxg6\" (UniqueName: \"kubernetes.io/projected/439ce07a-ea3d-4b97-a8ed-7655a840c873-kube-api-access-ndxg6\") pod \"439ce07a-ea3d-4b97-a8ed-7655a840c873\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " Oct 30 05:15:21.695312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.695236 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-tmp-dir\") pod \"439ce07a-ea3d-4b97-a8ed-7655a840c873\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " Oct 30 05:15:21.695526 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.695363 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-home\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-home\") pod \"439ce07a-ea3d-4b97-a8ed-7655a840c873\" (UID: \"439ce07a-ea3d-4b97-a8ed-7655a840c873\") " Oct 30 05:15:21.695526 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.695362 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-model-cache" pod "439ce07a-ea3d-4b97-a8ed-7655a840c873" (UID: "439ce07a-ea3d-4b97-a8ed-7655a840c873"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:21.695651 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.695590 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-home" pod "439ce07a-ea3d-4b97-a8ed-7655a840c873" (UID: "439ce07a-ea3d-4b97-a8ed-7655a840c873"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:21.695651 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.695601 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-tmp-dir" pod "439ce07a-ea3d-4b97-a8ed-7655a840c873" (UID: "439ce07a-ea3d-4b97-a8ed-7655a840c873"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:21.695824 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.695800 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:21.695885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.695829 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:21.695885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.695867 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:21.697459 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.697434 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-dshm" pod "439ce07a-ea3d-4b97-a8ed-7655a840c873" (UID: "439ce07a-ea3d-4b97-a8ed-7655a840c873"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:21.697570 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.697506 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439ce07a-ea3d-4b97-a8ed-7655a840c873-tls-certs" pod "439ce07a-ea3d-4b97-a8ed-7655a840c873" (UID: "439ce07a-ea3d-4b97-a8ed-7655a840c873"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:15:21.697620 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.697567 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439ce07a-ea3d-4b97-a8ed-7655a840c873-kube-api-access-ndxg6" pod "439ce07a-ea3d-4b97-a8ed-7655a840c873" (UID: "439ce07a-ea3d-4b97-a8ed-7655a840c873"). InnerVolumeSpecName "kube-api-access-ndxg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:21.796994 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.796956 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439ce07a-ea3d-4b97-a8ed-7655a840c873-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:21.796994 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.796993 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439ce07a-ea3d-4b97-a8ed-7655a840c873-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:21.797192 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:21.797007 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndxg6\" (UniqueName: \"kubernetes.io/projected/439ce07a-ea3d-4b97-a8ed-7655a840c873-kube-api-access-ndxg6\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:22.195700 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:22.195669 4793 generic.go:356] "Generic (PLEG): container finished" podID="439ce07a-ea3d-4b97-a8ed-7655a840c873" containerID="0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12" exitCode=0 Oct 30 05:15:22.195882 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:22.195717 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" Oct 30 05:15:22.195882 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:22.195756 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" event={"ID":"439ce07a-ea3d-4b97-a8ed-7655a840c873","Type":"ContainerDied","Data":"0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12"} Oct 30 05:15:22.195882 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:22.195813 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs" event={"ID":"439ce07a-ea3d-4b97-a8ed-7655a840c873","Type":"ContainerDied","Data":"9e0bdf80162895a9b6eccf80c4e2d81b06540228b459a6a7a81ad4670e4406c1"} Oct 30 05:15:22.195882 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:22.195835 4793 scope.go:122] "RemoveContainer" containerID="0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12" Oct 30 05:15:22.207219 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:22.207084 4793 scope.go:122] "RemoveContainer" containerID="0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12" Oct 30 05:15:22.207459 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:22.207431 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12\": container with ID starting with 0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12 not found: ID does not exist" containerID="0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12" Oct 30 05:15:22.207572 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:22.207476 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12"} err="failed to get container status \"0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12\": rpc error: code = NotFound desc = could not find container \"0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12\": container with ID starting with 0294ea06b7e4970f8d683b84df821980a480bb94ba5b39a29f59f3870de62d12 not found: ID does not exist" Oct 30 05:15:22.245201 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:22.245168 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs"] Oct 30 05:15:22.256584 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:22.256557 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-36bbac/distinct-w0-36bbac-kserve-bcf9bc74d-ll2xs"] Oct 30 05:15:22.523850 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:22.523761 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" podUID="824a3b82-1a31-4f68-8d37-4fa51299579b" containerName="payload-pre-processing" containerID="cri-o://ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c" gracePeriod=25 Oct 30 05:15:22.845481 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:22.845378 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="439ce07a-ea3d-4b97-a8ed-7655a840c873" path="/var/lib/kubelet/pods/439ce07a-ea3d-4b97-a8ed-7655a840c873/volumes" Oct 30 05:15:24.365124 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:24.365077 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:25.281178 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.281119 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" podUID="f53f3359-41ba-40d7-9820-c2409a0a6753" containerName="main" containerID="cri-o://86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228" gracePeriod=45 Oct 30 05:15:25.288164 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.288132 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" podUID="c9e51d14-8150-4f8c-9172-845b3b99eca9" containerName="main" containerID="cri-o://51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f" gracePeriod=45 Oct 30 05:15:25.316510 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.316469 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" podUID="c9e51d14-8150-4f8c-9172-845b3b99eca9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": dial tcp 10.133.0.73:8000: connect: connection refused" Oct 30 05:15:25.546197 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.546140 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 05:15:25.571804 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.571784 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 05:15:25.633243 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633209 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-home\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-home\") pod \"c9e51d14-8150-4f8c-9172-845b3b99eca9\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " Oct 30 05:15:25.633429 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633261 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9e51d14-8150-4f8c-9172-845b3b99eca9-tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e51d14-8150-4f8c-9172-845b3b99eca9-tls-certs\") pod \"c9e51d14-8150-4f8c-9172-845b3b99eca9\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " Oct 30 05:15:25.633429 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633302 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e51d14-8150-4f8c-9172-845b3b99eca9-kube-api-access-kbvwq\" (UniqueName: \"kubernetes.io/projected/c9e51d14-8150-4f8c-9172-845b3b99eca9-kube-api-access-kbvwq\") pod \"c9e51d14-8150-4f8c-9172-845b3b99eca9\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " Oct 30 05:15:25.633429 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633345 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-tmp-dir\") pod \"f53f3359-41ba-40d7-9820-c2409a0a6753\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " Oct 30 05:15:25.633429 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633374 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-model-cache\") pod \"f53f3359-41ba-40d7-9820-c2409a0a6753\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " Oct 30 05:15:25.633701 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633480 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-tmp-dir\") pod \"c9e51d14-8150-4f8c-9172-845b3b99eca9\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " Oct 30 05:15:25.633701 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633504 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-home" pod "c9e51d14-8150-4f8c-9172-845b3b99eca9" (UID: "c9e51d14-8150-4f8c-9172-845b3b99eca9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:25.633701 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633562 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-model-cache\") pod \"c9e51d14-8150-4f8c-9172-845b3b99eca9\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " Oct 30 05:15:25.633701 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633633 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-dshm\") pod \"f53f3359-41ba-40d7-9820-c2409a0a6753\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " Oct 30 05:15:25.633701 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633671 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-home\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-home\") pod \"f53f3359-41ba-40d7-9820-c2409a0a6753\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " Oct 30 05:15:25.633993 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633702 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53f3359-41ba-40d7-9820-c2409a0a6753-kube-api-access-dn22v\" (UniqueName: \"kubernetes.io/projected/f53f3359-41ba-40d7-9820-c2409a0a6753-kube-api-access-dn22v\") pod \"f53f3359-41ba-40d7-9820-c2409a0a6753\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " Oct 30 05:15:25.633993 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633730 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-tmp-dir" pod "f53f3359-41ba-40d7-9820-c2409a0a6753" (UID: "f53f3359-41ba-40d7-9820-c2409a0a6753"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:25.633993 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633814 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f53f3359-41ba-40d7-9820-c2409a0a6753-tls-certs\" (UniqueName: \"kubernetes.io/secret/f53f3359-41ba-40d7-9820-c2409a0a6753-tls-certs\") pod \"f53f3359-41ba-40d7-9820-c2409a0a6753\" (UID: \"f53f3359-41ba-40d7-9820-c2409a0a6753\") " Oct 30 05:15:25.633993 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.633850 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-dshm\") pod \"c9e51d14-8150-4f8c-9172-845b3b99eca9\" (UID: \"c9e51d14-8150-4f8c-9172-845b3b99eca9\") " Oct 30 05:15:25.634281 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.634258 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:25.634499 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.634477 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-home" pod "f53f3359-41ba-40d7-9820-c2409a0a6753" (UID: "f53f3359-41ba-40d7-9820-c2409a0a6753"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:25.638331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.634757 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-model-cache" pod "f53f3359-41ba-40d7-9820-c2409a0a6753" (UID: "f53f3359-41ba-40d7-9820-c2409a0a6753"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:25.638331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.634990 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-model-cache" pod "c9e51d14-8150-4f8c-9172-845b3b99eca9" (UID: "c9e51d14-8150-4f8c-9172-845b3b99eca9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:25.638331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.636133 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e51d14-8150-4f8c-9172-845b3b99eca9-tls-certs" pod "c9e51d14-8150-4f8c-9172-845b3b99eca9" (UID: "c9e51d14-8150-4f8c-9172-845b3b99eca9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:15:25.638331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.637130 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-dshm" pod "c9e51d14-8150-4f8c-9172-845b3b99eca9" (UID: "c9e51d14-8150-4f8c-9172-845b3b99eca9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:25.638331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.637816 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53f3359-41ba-40d7-9820-c2409a0a6753-tls-certs" pod "f53f3359-41ba-40d7-9820-c2409a0a6753" (UID: "f53f3359-41ba-40d7-9820-c2409a0a6753"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:15:25.638331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.638074 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-tmp-dir" pod "c9e51d14-8150-4f8c-9172-845b3b99eca9" (UID: "c9e51d14-8150-4f8c-9172-845b3b99eca9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:25.638928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.638904 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-dshm" pod "f53f3359-41ba-40d7-9820-c2409a0a6753" (UID: "f53f3359-41ba-40d7-9820-c2409a0a6753"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:25.639494 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.639452 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e51d14-8150-4f8c-9172-845b3b99eca9-kube-api-access-kbvwq" pod "c9e51d14-8150-4f8c-9172-845b3b99eca9" (UID: "c9e51d14-8150-4f8c-9172-845b3b99eca9"). InnerVolumeSpecName "kube-api-access-kbvwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:25.640373 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.640347 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53f3359-41ba-40d7-9820-c2409a0a6753-kube-api-access-dn22v" pod "f53f3359-41ba-40d7-9820-c2409a0a6753" (UID: "f53f3359-41ba-40d7-9820-c2409a0a6753"). InnerVolumeSpecName "kube-api-access-dn22v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:25.735254 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.735226 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:25.735254 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.735251 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9e51d14-8150-4f8c-9172-845b3b99eca9-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:25.735468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.735263 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbvwq\" (UniqueName: \"kubernetes.io/projected/c9e51d14-8150-4f8c-9172-845b3b99eca9-kube-api-access-kbvwq\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:25.735468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.735273 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:25.735468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.735282 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:25.735468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.735290 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:25.735468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.735298 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:25.735468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.735305 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f53f3359-41ba-40d7-9820-c2409a0a6753-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:25.735468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.735312 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn22v\" (UniqueName: \"kubernetes.io/projected/f53f3359-41ba-40d7-9820-c2409a0a6753-kube-api-access-dn22v\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:25.735468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.735320 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f53f3359-41ba-40d7-9820-c2409a0a6753-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:25.735468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:25.735327 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9e51d14-8150-4f8c-9172-845b3b99eca9-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:26.224035 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.224002 4793 generic.go:356] "Generic (PLEG): container finished" podID="f53f3359-41ba-40d7-9820-c2409a0a6753" containerID="86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228" exitCode=0 Oct 30 05:15:26.224223 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.224046 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" Oct 30 05:15:26.224223 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.224091 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" event={"ID":"f53f3359-41ba-40d7-9820-c2409a0a6753","Type":"ContainerDied","Data":"86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228"} Oct 30 05:15:26.224223 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.224134 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f" event={"ID":"f53f3359-41ba-40d7-9820-c2409a0a6753","Type":"ContainerDied","Data":"42ce81d2531826c403cb622d930aad8d352e5af9bd167c53e6d5609afa05de4b"} Oct 30 05:15:26.224223 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.224155 4793 scope.go:122] "RemoveContainer" containerID="86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228" Oct 30 05:15:26.225741 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.225716 4793 generic.go:356] "Generic (PLEG): container finished" podID="c9e51d14-8150-4f8c-9172-845b3b99eca9" containerID="51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f" exitCode=0 Oct 30 05:15:26.225868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.225760 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" Oct 30 05:15:26.225868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.225768 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" event={"ID":"c9e51d14-8150-4f8c-9172-845b3b99eca9","Type":"ContainerDied","Data":"51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f"} Oct 30 05:15:26.225868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.225800 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz" event={"ID":"c9e51d14-8150-4f8c-9172-845b3b99eca9","Type":"ContainerDied","Data":"1f682632df335aa9cbe0e6e3337f8a3e3ec4c09555e67cdfe7d1c436bc0b747a"} Oct 30 05:15:26.234617 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.234602 4793 scope.go:122] "RemoveContainer" containerID="86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228" Oct 30 05:15:26.234870 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:26.234850 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228\": container with ID starting with 86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228 not found: ID does not exist" containerID="86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228" Oct 30 05:15:26.234929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.234877 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228"} err="failed to get container status \"86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228\": rpc error: code = NotFound desc = could not find container \"86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228\": container with ID starting with 86fbeb1e4873007861dff7b754961cf8d4a5f82ec51cb8f44ade456497b4f228 not found: ID does not exist" Oct 30 05:15:26.234929 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.234895 4793 scope.go:122] "RemoveContainer" containerID="51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f" Oct 30 05:15:26.244012 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.243995 4793 scope.go:122] "RemoveContainer" containerID="51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f" Oct 30 05:15:26.244280 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:26.244260 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f\": container with ID starting with 51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f not found: ID does not exist" containerID="51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f" Oct 30 05:15:26.244349 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.244290 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f"} err="failed to get container status \"51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f\": rpc error: code = NotFound desc = could not find container \"51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f\": container with ID starting with 51ed6f8a01e6a572e9c77cb62bbfb33c989bd550af83eff341430712301ae21f not found: ID does not exist" Oct 30 05:15:26.274038 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.274012 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz"] Oct 30 05:15:26.285326 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.285303 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/simulator-w3-b4708a-kserve-57bf46994d-28mgz"] Oct 30 05:15:26.311429 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.311400 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f"] Oct 30 05:15:26.320470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.320450 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b4708a/premium-w3-b4708a-kserve-5c58db468f-pg86f"] Oct 30 05:15:26.845631 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.845597 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e51d14-8150-4f8c-9172-845b3b99eca9" path="/var/lib/kubelet/pods/c9e51d14-8150-4f8c-9172-845b3b99eca9/volumes" Oct 30 05:15:26.846010 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:26.845998 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53f3359-41ba-40d7-9820-c2409a0a6753" path="/var/lib/kubelet/pods/f53f3359-41ba-40d7-9820-c2409a0a6753/volumes" Oct 30 05:15:27.029178 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:27.029135 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" podUID="f186039e-f254-4792-bea7-3607de3d4a7f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:15:27.029178 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:27.029151 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" podUID="73c14d0b-ca18-4835-94c4-aedc0c1ad8e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:15:27.101918 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:27.101827 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:28.562837 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:28.562798 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:28.937481 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:28.937410 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:29.958667 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:29.958626 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" podUID="824a3b82-1a31-4f68-8d37-4fa51299579b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:15:30.265497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:30.264631 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:30.490451 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:30.490391 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:30.991863 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:30.991802 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:31.220610 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:31.220563 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w"] Oct 30 05:15:31.220903 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:31.220860 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" podUID="f4f3ab06-b4e7-4778-83f6-c43482adb47a" containerName="istio-proxy" containerID="cri-o://28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4" gracePeriod=30 Oct 30 05:15:31.390954 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:31.390863 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:33.021641 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.021603 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:33.222718 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.222692 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" Oct 30 05:15:33.272053 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.271983 4793 generic.go:356] "Generic (PLEG): container finished" podID="1c630cf4-f669-4563-8483-f9f367841185" containerID="3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2" exitCode=137 Oct 30 05:15:33.272053 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.272034 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" Oct 30 05:15:33.272265 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.272070 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" event={"ID":"1c630cf4-f669-4563-8483-f9f367841185","Type":"ContainerDied","Data":"3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2"} Oct 30 05:15:33.272265 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.272132 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz" event={"ID":"1c630cf4-f669-4563-8483-f9f367841185","Type":"ContainerDied","Data":"91025bbd98510df8e217fdca0a77006309834325ee2552c9169f1b04a7ebc199"} Oct 30 05:15:33.272265 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.272164 4793 scope.go:122] "RemoveContainer" containerID="3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2" Oct 30 05:15:33.283214 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.283192 4793 scope.go:122] "RemoveContainer" containerID="7939722a4ad8b34eb58c03b756d2626fd78feb2d9ee6873ac531bbc2cfbf7954" Oct 30 05:15:33.294006 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.293976 4793 scope.go:122] "RemoveContainer" containerID="3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2" Oct 30 05:15:33.294291 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:33.294272 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2\": container with ID starting with 3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2 not found: ID does not exist" containerID="3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2" Oct 30 05:15:33.294376 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.294299 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2"} err="failed to get container status \"3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2\": rpc error: code = NotFound desc = could not find container \"3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2\": container with ID starting with 3587aaf8fee9934e657b55660845833b0743ac6f687132dd36b6f2ef9f591ce2 not found: ID does not exist" Oct 30 05:15:33.294376 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.294318 4793 scope.go:122] "RemoveContainer" containerID="7939722a4ad8b34eb58c03b756d2626fd78feb2d9ee6873ac531bbc2cfbf7954" Oct 30 05:15:33.294662 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:33.294642 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7939722a4ad8b34eb58c03b756d2626fd78feb2d9ee6873ac531bbc2cfbf7954\": container with ID starting with 7939722a4ad8b34eb58c03b756d2626fd78feb2d9ee6873ac531bbc2cfbf7954 not found: ID does not exist" containerID="7939722a4ad8b34eb58c03b756d2626fd78feb2d9ee6873ac531bbc2cfbf7954" Oct 30 05:15:33.294735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.294667 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7939722a4ad8b34eb58c03b756d2626fd78feb2d9ee6873ac531bbc2cfbf7954"} err="failed to get container status \"7939722a4ad8b34eb58c03b756d2626fd78feb2d9ee6873ac531bbc2cfbf7954\": rpc error: code = NotFound desc = could not find container \"7939722a4ad8b34eb58c03b756d2626fd78feb2d9ee6873ac531bbc2cfbf7954\": container with ID starting with 7939722a4ad8b34eb58c03b756d2626fd78feb2d9ee6873ac531bbc2cfbf7954 not found: ID does not exist" Oct 30 05:15:33.302636 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.302610 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c630cf4-f669-4563-8483-f9f367841185-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c630cf4-f669-4563-8483-f9f367841185-maas-api-service-ca\") pod \"1c630cf4-f669-4563-8483-f9f367841185\" (UID: \"1c630cf4-f669-4563-8483-f9f367841185\") " Oct 30 05:15:33.302800 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.302783 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c630cf4-f669-4563-8483-f9f367841185-kube-api-access-dbb2w\" (UniqueName: \"kubernetes.io/projected/1c630cf4-f669-4563-8483-f9f367841185-kube-api-access-dbb2w\") pod \"1c630cf4-f669-4563-8483-f9f367841185\" (UID: \"1c630cf4-f669-4563-8483-f9f367841185\") " Oct 30 05:15:33.303080 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.303052 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c630cf4-f669-4563-8483-f9f367841185-maas-api-service-ca" pod "1c630cf4-f669-4563-8483-f9f367841185" (UID: "1c630cf4-f669-4563-8483-f9f367841185"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:33.305243 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.305212 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c630cf4-f669-4563-8483-f9f367841185-kube-api-access-dbb2w" pod "1c630cf4-f669-4563-8483-f9f367841185" (UID: "1c630cf4-f669-4563-8483-f9f367841185"). InnerVolumeSpecName "kube-api-access-dbb2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:33.322872 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.322844 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:33.404231 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.404195 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c630cf4-f669-4563-8483-f9f367841185-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:33.404231 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.404221 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbb2w\" (UniqueName: \"kubernetes.io/projected/1c630cf4-f669-4563-8483-f9f367841185-kube-api-access-dbb2w\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:33.622746 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.622712 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz"] Oct 30 05:15:33.624910 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.624889 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-495ebe-29888955-gplbz"] Oct 30 05:15:33.796372 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:33.796333 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:34.165515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:34.165471 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:34.843545 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:34.843513 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c630cf4-f669-4563-8483-f9f367841185" path="/var/lib/kubelet/pods/1c630cf4-f669-4563-8483-f9f367841185/volumes" Oct 30 05:15:35.319252 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:35.319217 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-c4zkg_8af2e200-3999-4f9b-a10c-1407bef727e3/manager/0.log" Oct 30 05:15:35.518119 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:35.518087 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pg5vv_12b6686c-fc28-4cc0-9c53-252a4e8a02bd/manager/0.log" Oct 30 05:15:35.976895 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:35.976860 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-tcmbg_9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793/manager/0.log" Oct 30 05:15:36.594493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.594467 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 05:15:36.634917 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.634892 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-token\" (UniqueName: \"kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-token\") pod \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " Oct 30 05:15:36.635071 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.634930 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-socket\") pod \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " Oct 30 05:15:36.635071 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.634956 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-kube-api-access-8sv5c\" (UniqueName: \"kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-kube-api-access-8sv5c\") pod \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " Oct 30 05:15:36.635071 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.634974 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-podinfo\") pod \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " Oct 30 05:15:36.635071 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.635023 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-data\") pod \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " Oct 30 05:15:36.635071 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.635064 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istiod-ca-cert\") pod \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " Oct 30 05:15:36.635356 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.635095 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-certs\") pod \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " Oct 30 05:15:36.635356 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.635172 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-credential-socket\") pod \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " Oct 30 05:15:36.635356 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.635173 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-socket" pod "f4f3ab06-b4e7-4778-83f6-c43482adb47a" (UID: "f4f3ab06-b4e7-4778-83f6-c43482adb47a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:36.635356 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.635204 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-envoy\") pod \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\" (UID: \"f4f3ab06-b4e7-4778-83f6-c43482adb47a\") " Oct 30 05:15:36.635645 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.635456 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-certs" pod "f4f3ab06-b4e7-4778-83f6-c43482adb47a" (UID: "f4f3ab06-b4e7-4778-83f6-c43482adb47a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:36.635645 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.635471 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-credential-socket" pod "f4f3ab06-b4e7-4778-83f6-c43482adb47a" (UID: "f4f3ab06-b4e7-4778-83f6-c43482adb47a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:36.635645 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.635551 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:36.635645 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.635574 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:36.635867 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.635659 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-data" pod "f4f3ab06-b4e7-4778-83f6-c43482adb47a" (UID: "f4f3ab06-b4e7-4778-83f6-c43482adb47a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:36.635867 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.635784 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istiod-ca-cert" pod "f4f3ab06-b4e7-4778-83f6-c43482adb47a" (UID: "f4f3ab06-b4e7-4778-83f6-c43482adb47a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:36.637297 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.637234 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-kube-api-access-8sv5c" pod "f4f3ab06-b4e7-4778-83f6-c43482adb47a" (UID: "f4f3ab06-b4e7-4778-83f6-c43482adb47a"). InnerVolumeSpecName "kube-api-access-8sv5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:36.637297 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.637256 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-podinfo" pod "f4f3ab06-b4e7-4778-83f6-c43482adb47a" (UID: "f4f3ab06-b4e7-4778-83f6-c43482adb47a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:15:36.637518 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.637393 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-envoy" pod "f4f3ab06-b4e7-4778-83f6-c43482adb47a" (UID: "f4f3ab06-b4e7-4778-83f6-c43482adb47a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:15:36.637518 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.637489 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-token" pod "f4f3ab06-b4e7-4778-83f6-c43482adb47a" (UID: "f4f3ab06-b4e7-4778-83f6-c43482adb47a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:36.736048 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.736016 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:36.736048 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.736044 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:36.736048 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.736054 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:36.736262 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.736063 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:36.736262 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.736073 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sv5c\" (UniqueName: \"kubernetes.io/projected/f4f3ab06-b4e7-4778-83f6-c43482adb47a-kube-api-access-8sv5c\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:36.736262 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.736081 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:36.736262 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:36.736089 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4f3ab06-b4e7-4778-83f6-c43482adb47a-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:37.029567 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:37.029527 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" podUID="73c14d0b-ca18-4835-94c4-aedc0c1ad8e7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:15:37.029567 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:37.029547 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" podUID="f186039e-f254-4792-bea7-3607de3d4a7f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:15:37.296621 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:37.296536 4793 generic.go:356] "Generic (PLEG): container finished" podID="f4f3ab06-b4e7-4778-83f6-c43482adb47a" containerID="28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4" exitCode=0 Oct 30 05:15:37.296621 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:37.296582 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" Oct 30 05:15:37.296838 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:37.296614 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" event={"ID":"f4f3ab06-b4e7-4778-83f6-c43482adb47a","Type":"ContainerDied","Data":"28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4"} Oct 30 05:15:37.296838 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:37.296669 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w" event={"ID":"f4f3ab06-b4e7-4778-83f6-c43482adb47a","Type":"ContainerDied","Data":"693afdc0c4f6c28566ab6023792651535bdf93aa7afd84da91689f3ab1e8611d"} Oct 30 05:15:37.296838 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:37.296692 4793 scope.go:122] "RemoveContainer" containerID="28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4" Oct 30 05:15:37.306337 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:37.306318 4793 scope.go:122] "RemoveContainer" containerID="28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4" Oct 30 05:15:37.306622 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:37.306602 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4\": container with ID starting with 28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4 not found: ID does not exist" containerID="28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4" Oct 30 05:15:37.306698 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:37.306633 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4"} err="failed to get container status \"28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4\": rpc error: code = NotFound desc = could not find container \"28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4\": container with ID starting with 28fbdb3b58737078c1b13ed2672588e29cca8a8d8ea079688f21dec0950beed4 not found: ID does not exist" Oct 30 05:15:37.357825 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:37.357800 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w"] Oct 30 05:15:37.369677 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:37.369654 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b4708a-openshift-default-f9f6fc547-z9b7w"] Oct 30 05:15:37.397682 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:37.397656 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79d77bcfdb-6hwnl_a736cf03-a95d-4e46-9358-7b8e6bba483b/manager/0.log" Oct 30 05:15:38.844530 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:38.844488 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f3ab06-b4e7-4778-83f6-c43482adb47a" path="/var/lib/kubelet/pods/f4f3ab06-b4e7-4778-83f6-c43482adb47a/volumes" Oct 30 05:15:38.858932 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:38.858901 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pxxpk_66328127-fc0b-4f8a-9599-2a82c297712d/manager/0.log" Oct 30 05:15:39.958510 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:39.958466 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" podUID="824a3b82-1a31-4f68-8d37-4fa51299579b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:15:40.316442 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.316386 4793 generic.go:356] "Generic (PLEG): container finished" podID="f186039e-f254-4792-bea7-3607de3d4a7f" containerID="f989dc363e40c8c9888871988145afe8f7972bee92b8e0c080799992e007fa53" exitCode=0 Oct 30 05:15:40.316600 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.316476 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" event={"ID":"f186039e-f254-4792-bea7-3607de3d4a7f","Type":"ContainerDied","Data":"f989dc363e40c8c9888871988145afe8f7972bee92b8e0c080799992e007fa53"} Oct 30 05:15:40.316600 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.316546 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" event={"ID":"f186039e-f254-4792-bea7-3607de3d4a7f","Type":"ContainerDied","Data":"50dcda9d96d1daebb8bd03ae72b0f59639271595dd403dd6fb92feeac8e0208a"} Oct 30 05:15:40.316600 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.316576 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50dcda9d96d1daebb8bd03ae72b0f59639271595dd403dd6fb92feeac8e0208a" Oct 30 05:15:40.318226 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.318206 4793 generic.go:356] "Generic (PLEG): container finished" podID="73c14d0b-ca18-4835-94c4-aedc0c1ad8e7" containerID="ae3dee5aa118ee3dbcc4ccc2a67d9858a0d7825ed1482705d799877d15cfe1f3" exitCode=0 Oct 30 05:15:40.318357 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.318261 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" event={"ID":"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7","Type":"ContainerDied","Data":"ae3dee5aa118ee3dbcc4ccc2a67d9858a0d7825ed1482705d799877d15cfe1f3"} Oct 30 05:15:40.319974 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.319956 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" Oct 30 05:15:40.346958 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.346940 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" Oct 30 05:15:40.369328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.369307 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f186039e-f254-4792-bea7-3607de3d4a7f-kube-api-access-s95ps\" (UniqueName: \"kubernetes.io/projected/f186039e-f254-4792-bea7-3607de3d4a7f-kube-api-access-s95ps\") pod \"f186039e-f254-4792-bea7-3607de3d4a7f\" (UID: \"f186039e-f254-4792-bea7-3607de3d4a7f\") " Oct 30 05:15:40.369439 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.369352 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f186039e-f254-4792-bea7-3607de3d4a7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f186039e-f254-4792-bea7-3607de3d4a7f-plugins-config-volume\") pod \"f186039e-f254-4792-bea7-3607de3d4a7f\" (UID: \"f186039e-f254-4792-bea7-3607de3d4a7f\") " Oct 30 05:15:40.369763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.369740 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f186039e-f254-4792-bea7-3607de3d4a7f-plugins-config-volume" pod "f186039e-f254-4792-bea7-3607de3d4a7f" (UID: "f186039e-f254-4792-bea7-3607de3d4a7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:40.371465 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.371432 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f186039e-f254-4792-bea7-3607de3d4a7f-kube-api-access-s95ps" pod "f186039e-f254-4792-bea7-3607de3d4a7f" (UID: "f186039e-f254-4792-bea7-3607de3d4a7f"). InnerVolumeSpecName "kube-api-access-s95ps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:40.469925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.469896 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-plugins-config-volume\") pod \"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7\" (UID: \"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7\") " Oct 30 05:15:40.470071 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.470049 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-kube-api-access-v27lx\" (UniqueName: \"kubernetes.io/projected/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-kube-api-access-v27lx\") pod \"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7\" (UID: \"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7\") " Oct 30 05:15:40.470256 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.470238 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f186039e-f254-4792-bea7-3607de3d4a7f-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:40.470334 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.470261 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s95ps\" (UniqueName: \"kubernetes.io/projected/f186039e-f254-4792-bea7-3607de3d4a7f-kube-api-access-s95ps\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:40.470334 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.470298 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-plugins-config-volume" pod "73c14d0b-ca18-4835-94c4-aedc0c1ad8e7" (UID: "73c14d0b-ca18-4835-94c4-aedc0c1ad8e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:40.472126 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.472106 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-kube-api-access-v27lx" pod "73c14d0b-ca18-4835-94c4-aedc0c1ad8e7" (UID: "73c14d0b-ca18-4835-94c4-aedc0c1ad8e7"). InnerVolumeSpecName "kube-api-access-v27lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:40.571734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.571677 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v27lx\" (UniqueName: \"kubernetes.io/projected/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-kube-api-access-v27lx\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:40.571734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:40.571699 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:41.218727 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:41.218691 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:15:41.324802 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:41.324772 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" Oct 30 05:15:41.324978 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:41.324800 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q" event={"ID":"73c14d0b-ca18-4835-94c4-aedc0c1ad8e7","Type":"ContainerDied","Data":"c02f887a37129a07dc4a7d1113f6bd6aac485b350732e8099ecf1da95ae333d1"} Oct 30 05:15:41.324978 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:41.324858 4793 scope.go:122] "RemoveContainer" containerID="ae3dee5aa118ee3dbcc4ccc2a67d9858a0d7825ed1482705d799877d15cfe1f3" Oct 30 05:15:41.324978 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:41.324966 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg" Oct 30 05:15:41.375192 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:41.375163 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg"] Oct 30 05:15:41.377930 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:41.377909 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-36bbac-75df4744d4-k67rg"] Oct 30 05:15:41.410277 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:41.410251 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q"] Oct 30 05:15:41.414281 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:41.414257 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-36bbac-7479548695-5fp2q"] Oct 30 05:15:42.777201 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:42.777173 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" Oct 30 05:15:42.845863 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:42.845796 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c14d0b-ca18-4835-94c4-aedc0c1ad8e7" path="/var/lib/kubelet/pods/73c14d0b-ca18-4835-94c4-aedc0c1ad8e7/volumes" Oct 30 05:15:42.846190 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:42.846176 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f186039e-f254-4792-bea7-3607de3d4a7f" path="/var/lib/kubelet/pods/f186039e-f254-4792-bea7-3607de3d4a7f/volumes" Oct 30 05:15:42.891559 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:42.891526 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824a3b82-1a31-4f68-8d37-4fa51299579b-kube-api-access-z2xz6\" (UniqueName: \"kubernetes.io/projected/824a3b82-1a31-4f68-8d37-4fa51299579b-kube-api-access-z2xz6\") pod \"824a3b82-1a31-4f68-8d37-4fa51299579b\" (UID: \"824a3b82-1a31-4f68-8d37-4fa51299579b\") " Oct 30 05:15:42.891706 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:42.891582 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824a3b82-1a31-4f68-8d37-4fa51299579b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824a3b82-1a31-4f68-8d37-4fa51299579b-plugins-config-volume\") pod \"824a3b82-1a31-4f68-8d37-4fa51299579b\" (UID: \"824a3b82-1a31-4f68-8d37-4fa51299579b\") " Oct 30 05:15:42.892035 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:42.892009 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824a3b82-1a31-4f68-8d37-4fa51299579b-plugins-config-volume" pod "824a3b82-1a31-4f68-8d37-4fa51299579b" (UID: "824a3b82-1a31-4f68-8d37-4fa51299579b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:15:42.893709 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:42.893685 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824a3b82-1a31-4f68-8d37-4fa51299579b-kube-api-access-z2xz6" pod "824a3b82-1a31-4f68-8d37-4fa51299579b" (UID: "824a3b82-1a31-4f68-8d37-4fa51299579b"). InnerVolumeSpecName "kube-api-access-z2xz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:15:42.992289 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:42.992262 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2xz6\" (UniqueName: \"kubernetes.io/projected/824a3b82-1a31-4f68-8d37-4fa51299579b-kube-api-access-z2xz6\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:42.992450 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:42.992292 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/824a3b82-1a31-4f68-8d37-4fa51299579b-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:15:43.340766 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:43.340732 4793 generic.go:356] "Generic (PLEG): container finished" podID="824a3b82-1a31-4f68-8d37-4fa51299579b" containerID="ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c" exitCode=0 Oct 30 05:15:43.340966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:43.340795 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" Oct 30 05:15:43.340966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:43.340808 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" event={"ID":"824a3b82-1a31-4f68-8d37-4fa51299579b","Type":"ContainerDied","Data":"ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c"} Oct 30 05:15:43.340966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:43.340868 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5" event={"ID":"824a3b82-1a31-4f68-8d37-4fa51299579b","Type":"ContainerDied","Data":"22e32948feac31eb3443cb9587559f4202aef0737916737e0bb44fc8c9e6fe04"} Oct 30 05:15:43.340966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:43.340893 4793 scope.go:122] "RemoveContainer" containerID="ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c" Oct 30 05:15:43.351188 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:43.351162 4793 scope.go:122] "RemoveContainer" containerID="ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c" Oct 30 05:15:43.351611 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:15:43.351573 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c\": container with ID starting with ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c not found: ID does not exist" containerID="ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c" Oct 30 05:15:43.351695 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:43.351619 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c"} err="failed to get container status \"ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c\": rpc error: code = NotFound desc = could not find container \"ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c\": container with ID starting with ef290ea6ed27a5a10181ef70ac29e18215848b21ed4dbb189079682f884df06c not found: ID does not exist" Oct 30 05:15:43.394182 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:43.394156 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5"] Oct 30 05:15:43.407204 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:43.407172 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b4708a-6c9475c6ff-pw7v5"] Oct 30 05:15:43.489761 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:43.489724 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:43.745872 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:43.745839 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:44.096231 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096115 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp"] Oct 30 05:15:44.096727 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096700 4793 cpu_manager.go:414] "containerMap: removing container" podUID="73c14d0b-ca18-4835-94c4-aedc0c1ad8e7" containerName="payload-pre-processing" Oct 30 05:15:44.096727 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096722 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c14d0b-ca18-4835-94c4-aedc0c1ad8e7" containerName="payload-pre-processing" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096750 4793 cpu_manager.go:414] "containerMap: removing container" podUID="824a3b82-1a31-4f68-8d37-4fa51299579b" containerName="payload-pre-processing" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096758 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824a3b82-1a31-4f68-8d37-4fa51299579b" containerName="payload-pre-processing" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096774 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f4f3ab06-b4e7-4778-83f6-c43482adb47a" containerName="istio-proxy" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096781 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f3ab06-b4e7-4778-83f6-c43482adb47a" containerName="istio-proxy" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096790 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1c630cf4-f669-4563-8483-f9f367841185" containerName="cleanup" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096797 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c630cf4-f669-4563-8483-f9f367841185" containerName="cleanup" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096814 4793 cpu_manager.go:414] "containerMap: removing container" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" containerName="cleanup" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096821 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" containerName="cleanup" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096832 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c9e51d14-8150-4f8c-9172-845b3b99eca9" containerName="main" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096837 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e51d14-8150-4f8c-9172-845b3b99eca9" containerName="main" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096846 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1c630cf4-f669-4563-8483-f9f367841185" containerName="cleanup" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096852 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c630cf4-f669-4563-8483-f9f367841185" containerName="cleanup" Oct 30 05:15:44.096858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096862 4793 cpu_manager.go:414] "containerMap: removing container" podUID="439ce07a-ea3d-4b97-a8ed-7655a840c873" containerName="main" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096869 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="439ce07a-ea3d-4b97-a8ed-7655a840c873" containerName="main" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096887 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3" containerName="revoke-keys" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096895 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3" containerName="revoke-keys" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096903 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f186039e-f254-4792-bea7-3607de3d4a7f" containerName="payload-processing" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096908 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f186039e-f254-4792-bea7-3607de3d4a7f" containerName="payload-processing" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096915 4793 cpu_manager.go:414] "containerMap: removing container" podUID="f53f3359-41ba-40d7-9820-c2409a0a6753" containerName="main" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096920 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53f3359-41ba-40d7-9820-c2409a0a6753" containerName="main" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096985 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f3ab06-b4e7-4778-83f6-c43482adb47a" containerName="istio-proxy" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.096994 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c630cf4-f669-4563-8483-f9f367841185" containerName="cleanup" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.097001 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e51d14-8150-4f8c-9172-845b3b99eca9" containerName="main" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.097009 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c630cf4-f669-4563-8483-f9f367841185" containerName="cleanup" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.097017 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="439ce07a-ea3d-4b97-a8ed-7655a840c873" containerName="main" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.097026 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3" containerName="revoke-keys" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.097032 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f186039e-f254-4792-bea7-3607de3d4a7f" containerName="payload-processing" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.097037 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53f3359-41ba-40d7-9820-c2409a0a6753" containerName="main" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.097042 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c14d0b-ca18-4835-94c4-aedc0c1ad8e7" containerName="payload-pre-processing" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.097048 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="4259a987-fbcc-4edf-867e-0f21fff10f9f" containerName="cleanup" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.097065 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="48947e23-be16-44b9-9c61-209679df23a9" containerName="cleanup" Oct 30 05:15:44.097387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.097072 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="824a3b82-1a31-4f68-8d37-4fa51299579b" containerName="payload-pre-processing" Oct 30 05:15:44.101515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.101495 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.115958 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.115937 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a566022b-openshift-default-dockercfg-2jx7l\"" Oct 30 05:15:44.131079 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.131053 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp"] Oct 30 05:15:44.204643 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.204593 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-podinfo\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.204776 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.204704 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-data\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.204776 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.204769 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-credential-socket\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.204919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.204850 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istiod-ca-cert\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.204919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.204894 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-socket\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.205034 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.204976 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-envoy\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.205034 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.205025 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn6wf\" (UniqueName: \"kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-kube-api-access-bn6wf\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.205152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.205064 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-token\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.205152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.205126 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-certs\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.254429 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.254385 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:44.305807 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.305777 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-data\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.305945 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.305832 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-credential-socket\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.305945 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.305855 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istiod-ca-cert\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.305945 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.305874 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-socket\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.305945 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.305906 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-envoy\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.305945 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.305937 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn6wf\" (UniqueName: \"kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-kube-api-access-bn6wf\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.306234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.305971 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-token\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.306234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.306000 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-certs\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.306234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.306038 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-podinfo\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.306234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.306174 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-data\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.306504 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.306482 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-credential-socket\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.306561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.306531 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-socket\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.306650 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.306629 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-certs\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.306692 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.306676 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istiod-ca-cert\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.308366 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.308349 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-envoy\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.308632 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.308614 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-podinfo\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.333593 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.333573 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn6wf\" (UniqueName: \"kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-kube-api-access-bn6wf\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.333721 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.333704 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-token\") pod \"e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.416830 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.416767 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:44.474118 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.474078 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:44.733759 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.733733 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp"] Oct 30 05:15:44.735481 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:15:44.735439 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70f3845_c325_4f1b_9caa_f8f582d0e16a.slice/crio-9cc3e194e803e15e1ec096f7a9ab10018f1fe135fe07ac5d7ac4d5d9cdbf6e44 WatchSource:0}: Error finding container 9cc3e194e803e15e1ec096f7a9ab10018f1fe135fe07ac5d7ac4d5d9cdbf6e44: Status 404 returned error can't find the container with id 9cc3e194e803e15e1ec096f7a9ab10018f1fe135fe07ac5d7ac4d5d9cdbf6e44 Oct 30 05:15:44.738068 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.738036 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:15:44.738156 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.738107 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:15:44.738156 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.738135 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:15:44.818025 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.817962 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:44.845811 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:44.845778 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824a3b82-1a31-4f68-8d37-4fa51299579b" path="/var/lib/kubelet/pods/824a3b82-1a31-4f68-8d37-4fa51299579b/volumes" Oct 30 05:15:45.355435 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:45.355392 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" event={"ID":"c70f3845-c325-4f1b-9caa-f8f582d0e16a","Type":"ContainerStarted","Data":"0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68"} Oct 30 05:15:45.355822 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:45.355474 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" event={"ID":"c70f3845-c325-4f1b-9caa-f8f582d0e16a","Type":"ContainerStarted","Data":"9cc3e194e803e15e1ec096f7a9ab10018f1fe135fe07ac5d7ac4d5d9cdbf6e44"} Oct 30 05:15:45.412655 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:45.412607 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" podStartSLOduration=1.412593664 podStartE2EDuration="1.412593664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:45.410322999 +0000 UTC m=+4668.696197765" watchObservedRunningTime="2026-10-30 05:15:45.412593664 +0000 UTC m=+4668.698468435" Oct 30 05:15:45.417894 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:45.417874 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:45.422793 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:45.422773 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:46.362109 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:46.362066 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:46.363029 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:46.363006 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:15:47.069237 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:47.069199 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:47.295150 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:47.295096 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:48.020211 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.020164 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:48.247343 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.247303 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:48.557245 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.557215 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq"] Oct 30 05:15:48.562029 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.562002 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.580219 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.580195 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq"] Oct 30 05:15:48.589924 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.589901 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:15:48.590098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.589927 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 05:15:48.590236 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.590181 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a566022b-serving-cert\"" Oct 30 05:15:48.593527 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.593490 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:15:48.593527 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.593524 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gw5bk\"" Oct 30 05:15:48.644840 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.644811 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-maas-api-tls\") pod \"maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.644997 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.644871 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-openshift-service-ca\") pod \"maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.644997 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.644978 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-metrics-certs\") pod \"maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.645128 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.645011 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2cp\" (UniqueName: \"kubernetes.io/projected/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-kube-api-access-9d2cp\") pod \"maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.746091 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.746053 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-metrics-certs\") pod \"maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.746289 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.746104 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2cp\" (UniqueName: \"kubernetes.io/projected/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-kube-api-access-9d2cp\") pod \"maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.746289 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.746179 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-maas-api-tls\") pod \"maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.746289 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.746243 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-openshift-service-ca\") pod \"maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.747063 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.747034 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-openshift-service-ca\") pod \"maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.749227 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.749195 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-metrics-certs\") pod \"maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.750804 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.750783 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-maas-api-tls\") pod \"maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.768306 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.768277 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:48.769272 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.769251 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2cp\" (UniqueName: \"kubernetes.io/projected/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-kube-api-access-9d2cp\") pod \"maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.876291 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.876222 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:48.931029 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.930998 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg"] Oct 30 05:15:48.936066 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.936042 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" Oct 30 05:15:48.956977 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.955757 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg"] Oct 30 05:15:48.961186 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.960999 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a566022b-dockercfg-4mfjz\"" Oct 30 05:15:48.961186 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.961061 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a566022b\"" Oct 30 05:15:48.981888 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:48.981861 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:49.052106 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:49.052080 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2288949f-eab0-4260-9a58-764895c1eb96-plugins-config-volume\") pod \"payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg\" (UID: \"2288949f-eab0-4260-9a58-764895c1eb96\") " pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" Oct 30 05:15:49.052511 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:49.052129 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlq5b\" (UniqueName: \"kubernetes.io/projected/2288949f-eab0-4260-9a58-764895c1eb96-kube-api-access-rlq5b\") pod \"payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg\" (UID: \"2288949f-eab0-4260-9a58-764895c1eb96\") " pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" Oct 30 05:15:49.153219 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:49.153153 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2288949f-eab0-4260-9a58-764895c1eb96-plugins-config-volume\") pod \"payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg\" (UID: \"2288949f-eab0-4260-9a58-764895c1eb96\") " pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" Oct 30 05:15:49.153219 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:49.153203 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlq5b\" (UniqueName: \"kubernetes.io/projected/2288949f-eab0-4260-9a58-764895c1eb96-kube-api-access-rlq5b\") pod \"payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg\" (UID: \"2288949f-eab0-4260-9a58-764895c1eb96\") " pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" Oct 30 05:15:49.153925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:49.153903 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2288949f-eab0-4260-9a58-764895c1eb96-plugins-config-volume\") pod \"payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg\" (UID: \"2288949f-eab0-4260-9a58-764895c1eb96\") " pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" Oct 30 05:15:49.175741 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:49.175718 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlq5b\" (UniqueName: \"kubernetes.io/projected/2288949f-eab0-4260-9a58-764895c1eb96-kube-api-access-rlq5b\") pod \"payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg\" (UID: \"2288949f-eab0-4260-9a58-764895c1eb96\") " pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" Oct 30 05:15:49.198298 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:49.198272 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq"] Oct 30 05:15:49.201967 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:15:49.201933 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a1851d_1ac7_4b62_b63c_1c5ae9325b9b.slice/crio-2c239ca4823f1a6c771a87c19aacd72b897c193a88059947af0220b41358aeb9 WatchSource:0}: Error finding container 2c239ca4823f1a6c771a87c19aacd72b897c193a88059947af0220b41358aeb9: Status 404 returned error can't find the container with id 2c239ca4823f1a6c771a87c19aacd72b897c193a88059947af0220b41358aeb9 Oct 30 05:15:49.204051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:49.204029 4793 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:15:49.244688 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:49.244663 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:49.261105 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:49.261082 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" Oct 30 05:15:49.380904 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:49.380865 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" event={"ID":"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b","Type":"ContainerStarted","Data":"2c239ca4823f1a6c771a87c19aacd72b897c193a88059947af0220b41358aeb9"} Oct 30 05:15:49.560091 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:49.560058 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg"] Oct 30 05:15:49.561599 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:15:49.561561 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2288949f_eab0_4260_9a58_764895c1eb96.slice/crio-8e7d602fccce8565d0ac54c08fb7312f039d803d7473061348a3e10fb4027fe5 WatchSource:0}: Error finding container 8e7d602fccce8565d0ac54c08fb7312f039d803d7473061348a3e10fb4027fe5: Status 404 returned error can't find the container with id 8e7d602fccce8565d0ac54c08fb7312f039d803d7473061348a3e10fb4027fe5 Oct 30 05:15:50.390145 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:50.390094 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" event={"ID":"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b","Type":"ContainerStarted","Data":"e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c"} Oct 30 05:15:50.390703 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:50.390408 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:50.392047 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:50.392020 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" event={"ID":"2288949f-eab0-4260-9a58-764895c1eb96","Type":"ContainerStarted","Data":"4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b"} Oct 30 05:15:50.392047 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:50.392059 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" event={"ID":"2288949f-eab0-4260-9a58-764895c1eb96","Type":"ContainerStarted","Data":"8e7d602fccce8565d0ac54c08fb7312f039d803d7473061348a3e10fb4027fe5"} Oct 30 05:15:50.392278 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:50.392162 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" Oct 30 05:15:50.433165 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:50.433119 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" podStartSLOduration=2.042981871 podStartE2EDuration="2.433105366s" totalImagesPullingTime="390.123495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:50.431342158 +0000 UTC m=+4673.717216939" watchObservedRunningTime="2026-10-30 05:15:50.433105366 +0000 UTC m=+4673.718980141" Oct 30 05:15:50.473598 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:50.473544 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" podStartSLOduration=2.47352799 podStartE2EDuration="2.47352799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:15:50.472932975 +0000 UTC m=+4673.758807752" watchObservedRunningTime="2026-10-30 05:15:50.47352799 +0000 UTC m=+4673.759402768" Oct 30 05:15:50.616321 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:50.616282 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:52.034841 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:52.034803 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:52.405865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:52.405759 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:54.107632 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:54.107592 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:55.348388 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:55.348356 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:55.814373 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:55.814329 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:56.404320 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:56.404296 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:15:57.422245 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:57.422206 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:59.209179 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:59.209143 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:59.622772 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:59.622687 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:15:59.682865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:15:59.682835 4793 scope.go:122] "RemoveContainer" containerID="f989dc363e40c8c9888871988145afe8f7972bee92b8e0c080799992e007fa53" Oct 30 05:16:00.908714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:00.908677 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:01.400768 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:01.400736 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" Oct 30 05:16:02.405573 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:02.405538 4793 patch_prober.go:29] interesting pod/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:16:02.405989 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:02.405612 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:16:02.638887 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:02.638848 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:03.465997 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:03.465965 4793 generic.go:356] "Generic (PLEG): container finished" podID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerID="0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68" exitCode=0 Oct 30 05:16:03.466367 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:03.466035 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" event={"ID":"c70f3845-c325-4f1b-9caa-f8f582d0e16a","Type":"ContainerDied","Data":"0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68"} Oct 30 05:16:03.466447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:03.466412 4793 scope.go:122] "RemoveContainer" containerID="0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68" Oct 30 05:16:03.467611 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:03.467579 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:16:03.467706 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:03.467649 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:16:03.467706 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:03.467686 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:16:04.477488 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:04.475516 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:04.482685 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:04.482649 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" event={"ID":"c70f3845-c325-4f1b-9caa-f8f582d0e16a","Type":"ContainerStarted","Data":"6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca"} Oct 30 05:16:05.417647 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:05.417616 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:16:05.417923 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:05.417899 4793 patch_prober.go:29] interesting pod/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Oct 30 05:16:05.418018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:05.417954 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Oct 30 05:16:06.279363 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:06.279325 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:06.417468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:06.417436 4793 patch_prober.go:29] interesting pod/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Oct 30 05:16:06.417632 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:06.417488 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Oct 30 05:16:07.418040 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:07.417997 4793 patch_prober.go:29] interesting pod/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Oct 30 05:16:07.418504 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:07.418059 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Oct 30 05:16:07.910993 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:07.910953 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:08.418048 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:08.418013 4793 patch_prober.go:29] interesting pod/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" start-of-body= Oct 30 05:16:08.418486 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:08.418064 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.4:15021/healthz/ready\": dial tcp 10.133.1.4:15021: connect: connection refused" Oct 30 05:16:09.456298 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:09.456262 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:16:09.457089 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:09.457064 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:16:09.458446 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:09.458407 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:16:09.530572 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:09.530540 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:11.035825 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:11.035792 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:11.218249 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:16:11.218216 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:16:11.265010 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:11.264975 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:12.824249 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:12.824216 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:13.308806 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:13.308766 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg"] Oct 30 05:16:13.552815 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:13.552786 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:13.757966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:13.757938 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:14.412105 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:14.412060 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:15.629012 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:15.628974 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq"] Oct 30 05:16:15.629518 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:15.629278 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" podUID="10a1851d-1ac7-4b62-b63c-1c5ae9325b9b" containerName="maas-api" containerID="cri-o://e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c" gracePeriod=30 Oct 30 05:16:15.838785 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:15.838747 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:15.886510 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:15.886445 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:16:15.944531 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:15.944499 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-maas-api-tls\") pod \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " Oct 30 05:16:15.944714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:15.944550 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-openshift-service-ca\") pod \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " Oct 30 05:16:15.944714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:15.944618 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-kube-api-access-9d2cp\" (UniqueName: \"kubernetes.io/projected/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-kube-api-access-9d2cp\") pod \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " Oct 30 05:16:15.944714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:15.944642 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-metrics-certs\") pod \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\" (UID: \"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b\") " Oct 30 05:16:15.945196 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:15.945164 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-openshift-service-ca" pod "10a1851d-1ac7-4b62-b63c-1c5ae9325b9b" (UID: "10a1851d-1ac7-4b62-b63c-1c5ae9325b9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:16:15.946876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:15.946852 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-kube-api-access-9d2cp" pod "10a1851d-1ac7-4b62-b63c-1c5ae9325b9b" (UID: "10a1851d-1ac7-4b62-b63c-1c5ae9325b9b"). InnerVolumeSpecName "kube-api-access-9d2cp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:15.947048 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:15.947023 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-metrics-certs" pod "10a1851d-1ac7-4b62-b63c-1c5ae9325b9b" (UID: "10a1851d-1ac7-4b62-b63c-1c5ae9325b9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:16:15.947048 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:15.947040 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-maas-api-tls" pod "10a1851d-1ac7-4b62-b63c-1c5ae9325b9b" (UID: "10a1851d-1ac7-4b62-b63c-1c5ae9325b9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:16:16.045924 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.045893 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:16.045924 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.045920 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:16.045924 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.045930 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2cp\" (UniqueName: \"kubernetes.io/projected/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-kube-api-access-9d2cp\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:16.046161 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.045938 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:16.307440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.307382 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:16.523501 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.523465 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:16.550837 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.550799 4793 generic.go:356] "Generic (PLEG): container finished" podID="10a1851d-1ac7-4b62-b63c-1c5ae9325b9b" containerID="e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c" exitCode=0 Oct 30 05:16:16.551011 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.550859 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" Oct 30 05:16:16.551011 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.550887 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" event={"ID":"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b","Type":"ContainerDied","Data":"e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c"} Oct 30 05:16:16.551011 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.550970 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq" event={"ID":"10a1851d-1ac7-4b62-b63c-1c5ae9325b9b","Type":"ContainerDied","Data":"2c239ca4823f1a6c771a87c19aacd72b897c193a88059947af0220b41358aeb9"} Oct 30 05:16:16.551011 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.550999 4793 scope.go:122] "RemoveContainer" containerID="e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c" Oct 30 05:16:16.561050 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.561030 4793 scope.go:122] "RemoveContainer" containerID="e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c" Oct 30 05:16:16.561299 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:16:16.561281 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c\": container with ID starting with e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c not found: ID does not exist" containerID="e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c" Oct 30 05:16:16.561357 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.561306 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c"} err="failed to get container status \"e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c\": rpc error: code = NotFound desc = could not find container \"e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c\": container with ID starting with e22d8a8c1ce1f590cefe5569a3d14bc82e9d63c3bc6c3cf5c4146340f3e8010c not found: ID does not exist" Oct 30 05:16:16.601279 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.601253 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq"] Oct 30 05:16:16.612625 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.612601 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a566022b-5dcc9776c-bk4fq"] Oct 30 05:16:16.844003 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:16.843926 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a1851d-1ac7-4b62-b63c-1c5ae9325b9b" path="/var/lib/kubelet/pods/10a1851d-1ac7-4b62-b63c-1c5ae9325b9b/volumes" Oct 30 05:16:18.309394 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.309350 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" podUID="2288949f-eab0-4260-9a58-764895c1eb96" containerName="payload-processing" containerID="cri-o://4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b" gracePeriod=25 Oct 30 05:16:18.552527 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.552503 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" Oct 30 05:16:18.566246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.566177 4793 generic.go:356] "Generic (PLEG): container finished" podID="2288949f-eab0-4260-9a58-764895c1eb96" containerID="4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b" exitCode=0 Oct 30 05:16:18.566246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.566216 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" event={"ID":"2288949f-eab0-4260-9a58-764895c1eb96","Type":"ContainerDied","Data":"4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b"} Oct 30 05:16:18.566467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.566278 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" Oct 30 05:16:18.566467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.566289 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg" event={"ID":"2288949f-eab0-4260-9a58-764895c1eb96","Type":"ContainerDied","Data":"8e7d602fccce8565d0ac54c08fb7312f039d803d7473061348a3e10fb4027fe5"} Oct 30 05:16:18.566467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.566320 4793 scope.go:122] "RemoveContainer" containerID="4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b" Oct 30 05:16:18.574940 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.574923 4793 scope.go:122] "RemoveContainer" containerID="4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b" Oct 30 05:16:18.575239 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:16:18.575221 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b\": container with ID starting with 4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b not found: ID does not exist" containerID="4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b" Oct 30 05:16:18.575291 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.575247 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b"} err="failed to get container status \"4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b\": rpc error: code = NotFound desc = could not find container \"4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b\": container with ID starting with 4193ab79fff5926de5c6d18ebfccdddb85d8299d6cb037923c9b1acc28e77c0b not found: ID does not exist" Oct 30 05:16:18.669537 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.669512 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2288949f-eab0-4260-9a58-764895c1eb96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2288949f-eab0-4260-9a58-764895c1eb96-plugins-config-volume\") pod \"2288949f-eab0-4260-9a58-764895c1eb96\" (UID: \"2288949f-eab0-4260-9a58-764895c1eb96\") " Oct 30 05:16:18.669679 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.669583 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2288949f-eab0-4260-9a58-764895c1eb96-kube-api-access-rlq5b\" (UniqueName: \"kubernetes.io/projected/2288949f-eab0-4260-9a58-764895c1eb96-kube-api-access-rlq5b\") pod \"2288949f-eab0-4260-9a58-764895c1eb96\" (UID: \"2288949f-eab0-4260-9a58-764895c1eb96\") " Oct 30 05:16:18.669950 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.669926 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2288949f-eab0-4260-9a58-764895c1eb96-plugins-config-volume" pod "2288949f-eab0-4260-9a58-764895c1eb96" (UID: "2288949f-eab0-4260-9a58-764895c1eb96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:16:18.671650 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.671627 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2288949f-eab0-4260-9a58-764895c1eb96-kube-api-access-rlq5b" pod "2288949f-eab0-4260-9a58-764895c1eb96" (UID: "2288949f-eab0-4260-9a58-764895c1eb96"). InnerVolumeSpecName "kube-api-access-rlq5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:18.771491 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.771464 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlq5b\" (UniqueName: \"kubernetes.io/projected/2288949f-eab0-4260-9a58-764895c1eb96-kube-api-access-rlq5b\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:18.771491 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.771492 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2288949f-eab0-4260-9a58-764895c1eb96-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:18.904044 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.904014 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg"] Oct 30 05:16:18.915499 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:18.915474 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a566022b-5cd659b5bb-7rzrg"] Oct 30 05:16:19.174069 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:19.173988 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp"] Oct 30 05:16:19.174252 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:19.174221 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerName="istio-proxy" containerID="cri-o://6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca" gracePeriod=30 Oct 30 05:16:19.288243 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:19.288210 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:19.512557 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:19.512521 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:20.851693 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:20.851653 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2288949f-eab0-4260-9a58-764895c1eb96" path="/var/lib/kubelet/pods/2288949f-eab0-4260-9a58-764895c1eb96/volumes" Oct 30 05:16:24.427180 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.427152 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:16:24.532009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.531915 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-certs\") pod \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " Oct 30 05:16:24.532009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.531969 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-token\" (UniqueName: \"kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-token\") pod \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " Oct 30 05:16:24.532009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532005 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-data\") pod \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " Oct 30 05:16:24.532299 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532046 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-socket\") pod \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " Oct 30 05:16:24.532299 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532073 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-envoy\") pod \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " Oct 30 05:16:24.532299 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532117 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-kube-api-access-bn6wf\" (UniqueName: \"kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-kube-api-access-bn6wf\") pod \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " Oct 30 05:16:24.532299 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532191 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-podinfo\") pod \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " Oct 30 05:16:24.532299 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532244 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istiod-ca-cert\") pod \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " Oct 30 05:16:24.532299 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532255 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-certs" pod "c70f3845-c325-4f1b-9caa-f8f582d0e16a" (UID: "c70f3845-c325-4f1b-9caa-f8f582d0e16a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:24.532635 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532305 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-credential-socket\") pod \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\" (UID: \"c70f3845-c325-4f1b-9caa-f8f582d0e16a\") " Oct 30 05:16:24.532635 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532568 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-socket" pod "c70f3845-c325-4f1b-9caa-f8f582d0e16a" (UID: "c70f3845-c325-4f1b-9caa-f8f582d0e16a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:24.532761 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532667 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:24.532761 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532687 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:24.532761 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532701 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-data" pod "c70f3845-c325-4f1b-9caa-f8f582d0e16a" (UID: "c70f3845-c325-4f1b-9caa-f8f582d0e16a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:24.532927 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.532819 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-credential-socket" pod "c70f3845-c325-4f1b-9caa-f8f582d0e16a" (UID: "c70f3845-c325-4f1b-9caa-f8f582d0e16a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:24.533097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.533053 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istiod-ca-cert" pod "c70f3845-c325-4f1b-9caa-f8f582d0e16a" (UID: "c70f3845-c325-4f1b-9caa-f8f582d0e16a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:16:24.534714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.534679 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-token" pod "c70f3845-c325-4f1b-9caa-f8f582d0e16a" (UID: "c70f3845-c325-4f1b-9caa-f8f582d0e16a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:24.534714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.534683 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-envoy" pod "c70f3845-c325-4f1b-9caa-f8f582d0e16a" (UID: "c70f3845-c325-4f1b-9caa-f8f582d0e16a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:16:24.534887 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.534872 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-podinfo" pod "c70f3845-c325-4f1b-9caa-f8f582d0e16a" (UID: "c70f3845-c325-4f1b-9caa-f8f582d0e16a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:16:24.535092 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.535067 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-kube-api-access-bn6wf" pod "c70f3845-c325-4f1b-9caa-f8f582d0e16a" (UID: "c70f3845-c325-4f1b-9caa-f8f582d0e16a"). InnerVolumeSpecName "kube-api-access-bn6wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:24.599455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.599396 4793 generic.go:356] "Generic (PLEG): container finished" podID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerID="6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca" exitCode=0 Oct 30 05:16:24.599616 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.599469 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" Oct 30 05:16:24.599616 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.599467 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" event={"ID":"c70f3845-c325-4f1b-9caa-f8f582d0e16a","Type":"ContainerDied","Data":"6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca"} Oct 30 05:16:24.599616 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.599539 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp" event={"ID":"c70f3845-c325-4f1b-9caa-f8f582d0e16a","Type":"ContainerDied","Data":"9cc3e194e803e15e1ec096f7a9ab10018f1fe135fe07ac5d7ac4d5d9cdbf6e44"} Oct 30 05:16:24.599616 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.599572 4793 scope.go:122] "RemoveContainer" containerID="6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca" Oct 30 05:16:24.613141 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.611521 4793 scope.go:122] "RemoveContainer" containerID="0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68" Oct 30 05:16:24.620483 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.620463 4793 scope.go:122] "RemoveContainer" containerID="6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca" Oct 30 05:16:24.620756 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:16:24.620736 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca\": container with ID starting with 6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca not found: ID does not exist" containerID="6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca" Oct 30 05:16:24.620827 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.620771 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca"} err="failed to get container status \"6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca\": rpc error: code = NotFound desc = could not find container \"6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca\": container with ID starting with 6930fd5a38760bdb1c0e5ea36c411d8287457eec692a9950bbe49c670ed8d7ca not found: ID does not exist" Oct 30 05:16:24.620827 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.620794 4793 scope.go:122] "RemoveContainer" containerID="0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68" Oct 30 05:16:24.621029 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:16:24.621012 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68\": container with ID starting with 0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68 not found: ID does not exist" containerID="0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68" Oct 30 05:16:24.621072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.621033 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68"} err="failed to get container status \"0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68\": rpc error: code = NotFound desc = could not find container \"0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68\": container with ID starting with 0fd71f5306166ff9f6d6138b9f53759782ff2883ead2a42f93f2447c68ab6d68 not found: ID does not exist" Oct 30 05:16:24.633841 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.633821 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:24.633841 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.633841 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:24.633980 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.633849 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:24.633980 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.633858 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn6wf\" (UniqueName: \"kubernetes.io/projected/c70f3845-c325-4f1b-9caa-f8f582d0e16a-kube-api-access-bn6wf\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:24.633980 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.633867 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:24.633980 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.633876 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c70f3845-c325-4f1b-9caa-f8f582d0e16a-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:24.633980 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.633884 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c70f3845-c325-4f1b-9caa-f8f582d0e16a-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:24.650822 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.650802 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp"] Oct 30 05:16:24.657071 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.657047 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a566022b-openshift-default-544fdb78c7-9jhgp"] Oct 30 05:16:24.845168 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:24.845092 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" path="/var/lib/kubelet/pods/c70f3845-c325-4f1b-9caa-f8f582d0e16a/volumes" Oct 30 05:16:26.244633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.244598 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f4559f84c-cb6r5"] Oct 30 05:16:26.245073 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.244953 4793 cpu_manager.go:414] "containerMap: removing container" podUID="10a1851d-1ac7-4b62-b63c-1c5ae9325b9b" containerName="maas-api" Oct 30 05:16:26.245073 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.244965 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a1851d-1ac7-4b62-b63c-1c5ae9325b9b" containerName="maas-api" Oct 30 05:16:26.245073 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.244977 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerName="istio-proxy" Oct 30 05:16:26.245073 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.244982 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerName="istio-proxy" Oct 30 05:16:26.245073 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.244994 4793 cpu_manager.go:414] "containerMap: removing container" podUID="2288949f-eab0-4260-9a58-764895c1eb96" containerName="payload-processing" Oct 30 05:16:26.245073 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.245000 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2288949f-eab0-4260-9a58-764895c1eb96" containerName="payload-processing" Oct 30 05:16:26.245073 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.245018 4793 cpu_manager.go:414] "containerMap: removing container" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerName="istio-proxy" Oct 30 05:16:26.245073 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.245023 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerName="istio-proxy" Oct 30 05:16:26.245073 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.245072 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a1851d-1ac7-4b62-b63c-1c5ae9325b9b" containerName="maas-api" Oct 30 05:16:26.245460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.245079 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerName="istio-proxy" Oct 30 05:16:26.245460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.245087 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="2288949f-eab0-4260-9a58-764895c1eb96" containerName="payload-processing" Oct 30 05:16:26.245460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.245096 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70f3845-c325-4f1b-9caa-f8f582d0e16a" containerName="istio-proxy" Oct 30 05:16:26.250779 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.250347 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:26.264893 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.264870 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 05:16:26.276585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.276562 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 05:16:26.276713 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.276607 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-j8pc2\"" Oct 30 05:16:26.276713 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.276647 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 05:16:26.280354 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.280332 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f4559f84c-cb6r5"] Oct 30 05:16:26.347084 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.347055 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-webhook-cert\") pod \"maas-controller-6f4559f84c-cb6r5\" (UID: \"cb490558-44fb-4262-a7ee-7318cca73d10\") " pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:26.347218 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.347095 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7f26\" (UniqueName: \"kubernetes.io/projected/cb490558-44fb-4262-a7ee-7318cca73d10-kube-api-access-p7f26\") pod \"maas-controller-6f4559f84c-cb6r5\" (UID: \"cb490558-44fb-4262-a7ee-7318cca73d10\") " pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:26.347218 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.347185 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-metrics-certs\") pod \"maas-controller-6f4559f84c-cb6r5\" (UID: \"cb490558-44fb-4262-a7ee-7318cca73d10\") " pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:26.448491 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.448463 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-webhook-cert\") pod \"maas-controller-6f4559f84c-cb6r5\" (UID: \"cb490558-44fb-4262-a7ee-7318cca73d10\") " pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:26.448627 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.448503 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7f26\" (UniqueName: \"kubernetes.io/projected/cb490558-44fb-4262-a7ee-7318cca73d10-kube-api-access-p7f26\") pod \"maas-controller-6f4559f84c-cb6r5\" (UID: \"cb490558-44fb-4262-a7ee-7318cca73d10\") " pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:26.448627 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.448542 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-metrics-certs\") pod \"maas-controller-6f4559f84c-cb6r5\" (UID: \"cb490558-44fb-4262-a7ee-7318cca73d10\") " pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:26.450767 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.450742 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-metrics-certs\") pod \"maas-controller-6f4559f84c-cb6r5\" (UID: \"cb490558-44fb-4262-a7ee-7318cca73d10\") " pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:26.450912 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.450891 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-webhook-cert\") pod \"maas-controller-6f4559f84c-cb6r5\" (UID: \"cb490558-44fb-4262-a7ee-7318cca73d10\") " pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:26.471363 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.471340 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7f26\" (UniqueName: \"kubernetes.io/projected/cb490558-44fb-4262-a7ee-7318cca73d10-kube-api-access-p7f26\") pod \"maas-controller-6f4559f84c-cb6r5\" (UID: \"cb490558-44fb-4262-a7ee-7318cca73d10\") " pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:26.563685 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.563629 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:26.797860 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.797828 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:26.857375 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:26.857352 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f4559f84c-cb6r5"] Oct 30 05:16:26.859710 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:16:26.859683 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb490558_44fb_4262_a7ee_7318cca73d10.slice/crio-9ae5f6ef1b3ce0086189a7b151935d8f771570e2b636016a6895f59705d4e360 WatchSource:0}: Error finding container 9ae5f6ef1b3ce0086189a7b151935d8f771570e2b636016a6895f59705d4e360: Status 404 returned error can't find the container with id 9ae5f6ef1b3ce0086189a7b151935d8f771570e2b636016a6895f59705d4e360 Oct 30 05:16:27.020069 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:27.020038 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:27.620713 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:27.620679 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4559f84c-cb6r5" event={"ID":"cb490558-44fb-4262-a7ee-7318cca73d10","Type":"ContainerStarted","Data":"9ae5f6ef1b3ce0086189a7b151935d8f771570e2b636016a6895f59705d4e360"} Oct 30 05:16:29.641200 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:29.641166 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4559f84c-cb6r5" event={"ID":"cb490558-44fb-4262-a7ee-7318cca73d10","Type":"ContainerStarted","Data":"a58583b3c5e00d6b0215364aec34a77b0624950c31fe5084bf497de7ffeb4e2b"} Oct 30 05:16:29.641605 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:29.641290 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:29.682995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:29.682935 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6f4559f84c-cb6r5" podStartSLOduration=1.836867914 podStartE2EDuration="3.682917322s" totalImagesPullingTime="1.846049408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:29.682386708 +0000 UTC m=+4712.968261482" watchObservedRunningTime="2026-10-30 05:16:29.682917322 +0000 UTC m=+4712.968792100" Oct 30 05:16:40.648256 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:40.648175 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:41.218368 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:16:41.218334 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:16:42.913055 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:42.913017 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:43.131259 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.131226 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:43.907022 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.906989 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg"] Oct 30 05:16:43.910339 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.910317 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:43.922097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.922075 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8"] Oct 30 05:16:43.925354 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.925315 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:43.931269 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.931209 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-vwdzb\"" Oct 30 05:16:43.937619 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.937598 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg"] Oct 30 05:16:43.948506 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.948484 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8"] Oct 30 05:16:43.988569 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.988544 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:43.988697 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.988576 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:43.988697 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.988602 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:43.988697 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.988655 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:43.988816 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.988752 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:43.988816 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.988781 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:43.988816 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.988799 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:43.988922 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.988816 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:43.988922 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.988866 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jmbf\" (UniqueName: \"kubernetes.io/projected/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-kube-api-access-4jmbf\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:43.988995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.988931 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:43.988995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.988968 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7qfs\" (UniqueName: \"kubernetes.io/projected/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-kube-api-access-r7qfs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:43.989066 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.988992 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:43.989066 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.989056 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:43.989133 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.989084 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:43.989166 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.989158 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:43.989205 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.989176 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:43.989205 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.989195 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:43.989278 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:43.989231 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.090520 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090491 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.090689 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090537 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7qfs\" (UniqueName: \"kubernetes.io/projected/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-kube-api-access-r7qfs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.090689 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090577 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.090689 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090595 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.090689 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090611 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.090889 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090692 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.090889 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090725 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.090889 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090753 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.090889 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090778 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.090889 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090827 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.090889 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090859 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.091285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090894 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.091285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090924 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.091285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090945 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.091285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090978 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.091285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.091136 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.091285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.091218 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.091681 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.090958 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.091681 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.091497 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.091681 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.091404 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.091681 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.091532 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.091681 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.091563 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.091681 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.091593 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jmbf\" (UniqueName: \"kubernetes.io/projected/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-kube-api-access-4jmbf\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.092013 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.091919 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.092013 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.091498 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.092136 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.092033 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.092136 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.092128 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.092227 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.092125 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.093309 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.093286 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.093773 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.093755 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.093855 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.093786 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.093919 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.093874 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.113224 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.113199 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.113352 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.113288 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.113697 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.113675 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7qfs\" (UniqueName: \"kubernetes.io/projected/9b3ff486-e88f-459b-a0b1-aa5d88c6660d-kube-api-access-r7qfs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-c99g8\" (UID: \"9b3ff486-e88f-459b-a0b1-aa5d88c6660d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.113778 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.113697 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jmbf\" (UniqueName: \"kubernetes.io/projected/84c295aa-6250-4bb0-bb4d-4fe24004a9f4-kube-api-access-4jmbf\") pod \"maas-default-gateway-openshift-default-7f694df8bc-h7nkg\" (UID: \"84c295aa-6250-4bb0-bb4d-4fe24004a9f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.223976 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.223934 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:44.237125 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.237105 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:44.548207 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.548186 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg"] Oct 30 05:16:44.550863 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.550835 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8"] Oct 30 05:16:44.550863 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:16:44.550853 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c295aa_6250_4bb0_bb4d_4fe24004a9f4.slice/crio-a387fc190b4dc1751d7fb144db99470164b5edfbee665251f70d31d121951fb5 WatchSource:0}: Error finding container a387fc190b4dc1751d7fb144db99470164b5edfbee665251f70d31d121951fb5: Status 404 returned error can't find the container with id a387fc190b4dc1751d7fb144db99470164b5edfbee665251f70d31d121951fb5 Oct 30 05:16:44.552976 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.552946 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:16:44.553079 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.553004 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:16:44.553079 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.553033 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:16:44.562013 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:16:44.561987 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3ff486_e88f_459b_a0b1_aa5d88c6660d.slice/crio-b87d6067ccae1d84fe2c9bf0426cd7e4dd409a7a6eeb518a39feb1076554d1da WatchSource:0}: Error finding container b87d6067ccae1d84fe2c9bf0426cd7e4dd409a7a6eeb518a39feb1076554d1da: Status 404 returned error can't find the container with id b87d6067ccae1d84fe2c9bf0426cd7e4dd409a7a6eeb518a39feb1076554d1da Oct 30 05:16:44.564007 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.563980 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:16:44.564097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.564035 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:16:44.564097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.564064 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:16:44.723498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.723455 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" event={"ID":"84c295aa-6250-4bb0-bb4d-4fe24004a9f4","Type":"ContainerStarted","Data":"bdb9c3337e1e60884ea6fd81b8a551d86b74b46d7d7a3e731afaa79d8ca83933"} Oct 30 05:16:44.723683 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.723524 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" event={"ID":"84c295aa-6250-4bb0-bb4d-4fe24004a9f4","Type":"ContainerStarted","Data":"a387fc190b4dc1751d7fb144db99470164b5edfbee665251f70d31d121951fb5"} Oct 30 05:16:44.725542 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.725515 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" event={"ID":"9b3ff486-e88f-459b-a0b1-aa5d88c6660d","Type":"ContainerStarted","Data":"e8ab9bc1bec102269a3475edd033a635c75eab3d395791b8aa2ecc8fae002fe5"} Oct 30 05:16:44.725678 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.725567 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" event={"ID":"9b3ff486-e88f-459b-a0b1-aa5d88c6660d","Type":"ContainerStarted","Data":"b87d6067ccae1d84fe2c9bf0426cd7e4dd409a7a6eeb518a39feb1076554d1da"} Oct 30 05:16:44.769047 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.768996 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podStartSLOduration=1.768983066 podStartE2EDuration="1.768983066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:44.766869866 +0000 UTC m=+4728.052744781" watchObservedRunningTime="2026-10-30 05:16:44.768983066 +0000 UTC m=+4728.054857842" Oct 30 05:16:44.811460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:44.811293 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" podStartSLOduration=1.8112753129999999 podStartE2EDuration="1.811275313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:44.809845682 +0000 UTC m=+4728.095720457" watchObservedRunningTime="2026-10-30 05:16:44.811275313 +0000 UTC m=+4728.097150088" Oct 30 05:16:45.225119 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:45.225087 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:45.227089 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:45.227061 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h7nkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 05:16:45.227252 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:45.227099 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podUID="84c295aa-6250-4bb0-bb4d-4fe24004a9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 05:16:45.237762 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:45.237731 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:45.238829 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:45.238809 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-c99g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Oct 30 05:16:45.238912 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:45.238849 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" podUID="9b3ff486-e88f-459b-a0b1-aa5d88c6660d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Oct 30 05:16:46.224652 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:46.224618 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h7nkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 05:16:46.224846 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:46.224668 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podUID="84c295aa-6250-4bb0-bb4d-4fe24004a9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 05:16:46.237917 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:46.237886 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-c99g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Oct 30 05:16:46.238269 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:46.237928 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" podUID="9b3ff486-e88f-459b-a0b1-aa5d88c6660d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Oct 30 05:16:47.224925 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:47.224887 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h7nkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 05:16:47.225149 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:47.224947 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podUID="84c295aa-6250-4bb0-bb4d-4fe24004a9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 05:16:47.237734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:47.237702 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-c99g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Oct 30 05:16:47.237888 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:47.237751 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" podUID="9b3ff486-e88f-459b-a0b1-aa5d88c6660d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Oct 30 05:16:48.225173 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:48.225137 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h7nkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 05:16:48.225733 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:48.225193 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podUID="84c295aa-6250-4bb0-bb4d-4fe24004a9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 05:16:48.237574 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:48.237544 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-c99g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Oct 30 05:16:48.237722 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:48.237592 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" podUID="9b3ff486-e88f-459b-a0b1-aa5d88c6660d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Oct 30 05:16:49.224923 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:49.224878 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h7nkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 05:16:49.225139 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:49.224933 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podUID="84c295aa-6250-4bb0-bb4d-4fe24004a9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 05:16:49.237701 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:49.237648 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-c99g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Oct 30 05:16:49.238056 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:49.237716 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" podUID="9b3ff486-e88f-459b-a0b1-aa5d88c6660d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Oct 30 05:16:50.225165 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:50.225123 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h7nkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 05:16:50.225489 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:50.225443 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podUID="84c295aa-6250-4bb0-bb4d-4fe24004a9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 05:16:50.237598 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:50.237569 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-c99g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Oct 30 05:16:50.237699 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:50.237618 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" podUID="9b3ff486-e88f-459b-a0b1-aa5d88c6660d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Oct 30 05:16:51.225373 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:51.225216 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h7nkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 05:16:51.225373 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:51.225281 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podUID="84c295aa-6250-4bb0-bb4d-4fe24004a9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 05:16:51.237672 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:51.237629 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-c99g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Oct 30 05:16:51.237853 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:51.237696 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" podUID="9b3ff486-e88f-459b-a0b1-aa5d88c6660d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Oct 30 05:16:53.224359 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:53.224316 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h7nkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:16:53.224862 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:53.224394 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podUID="84c295aa-6250-4bb0-bb4d-4fe24004a9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:16:53.239005 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:53.238972 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-c99g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:16:53.239150 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:53.239037 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" podUID="9b3ff486-e88f-459b-a0b1-aa5d88c6660d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:16:54.265639 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:54.265552 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h7nkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:16:54.265639 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:54.265615 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podUID="84c295aa-6250-4bb0-bb4d-4fe24004a9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:16:54.280592 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:54.280554 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-c99g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:16:54.280819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:54.280789 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" podUID="9b3ff486-e88f-459b-a0b1-aa5d88c6660d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:16:55.281835 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:55.281767 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-c99g8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:16:55.282500 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:55.282446 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" podUID="9b3ff486-e88f-459b-a0b1-aa5d88c6660d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:16:55.306534 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:55.306500 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h7nkg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:16:55.306678 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:55.306547 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podUID="84c295aa-6250-4bb0-bb4d-4fe24004a9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:16:55.340502 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:55.340376 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:55.341019 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:55.340997 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:55.341894 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:55.341875 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" Oct 30 05:16:55.453246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:55.453212 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:55.453696 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:55.453671 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:55.454763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:55.454726 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" Oct 30 05:16:58.621914 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.621875 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:58.643720 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.643693 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f4559f84c-cb6r5"] Oct 30 05:16:58.643932 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.643908 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6f4559f84c-cb6r5" podUID="cb490558-44fb-4262-a7ee-7318cca73d10" containerName="manager" containerID="cri-o://a58583b3c5e00d6b0215364aec34a77b0624950c31fe5084bf497de7ffeb4e2b" gracePeriod=30 Oct 30 05:16:58.876125 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.876101 4793 generic.go:356] "Generic (PLEG): container finished" podID="cb490558-44fb-4262-a7ee-7318cca73d10" containerID="a58583b3c5e00d6b0215364aec34a77b0624950c31fe5084bf497de7ffeb4e2b" exitCode=0 Oct 30 05:16:58.876241 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.876140 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4559f84c-cb6r5" event={"ID":"cb490558-44fb-4262-a7ee-7318cca73d10","Type":"ContainerDied","Data":"a58583b3c5e00d6b0215364aec34a77b0624950c31fe5084bf497de7ffeb4e2b"} Oct 30 05:16:58.876241 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.876176 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f4559f84c-cb6r5" event={"ID":"cb490558-44fb-4262-a7ee-7318cca73d10","Type":"ContainerDied","Data":"9ae5f6ef1b3ce0086189a7b151935d8f771570e2b636016a6895f59705d4e360"} Oct 30 05:16:58.876241 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.876192 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ae5f6ef1b3ce0086189a7b151935d8f771570e2b636016a6895f59705d4e360" Oct 30 05:16:58.887214 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.887194 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:58.938142 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.938110 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-metrics-certs\") pod \"cb490558-44fb-4262-a7ee-7318cca73d10\" (UID: \"cb490558-44fb-4262-a7ee-7318cca73d10\") " Oct 30 05:16:58.938279 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.938172 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-webhook-cert\") pod \"cb490558-44fb-4262-a7ee-7318cca73d10\" (UID: \"cb490558-44fb-4262-a7ee-7318cca73d10\") " Oct 30 05:16:58.938668 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.938328 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb490558-44fb-4262-a7ee-7318cca73d10-kube-api-access-p7f26\" (UniqueName: \"kubernetes.io/projected/cb490558-44fb-4262-a7ee-7318cca73d10-kube-api-access-p7f26\") pod \"cb490558-44fb-4262-a7ee-7318cca73d10\" (UID: \"cb490558-44fb-4262-a7ee-7318cca73d10\") " Oct 30 05:16:58.940838 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.940808 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-metrics-certs" pod "cb490558-44fb-4262-a7ee-7318cca73d10" (UID: "cb490558-44fb-4262-a7ee-7318cca73d10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:16:58.941023 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.940978 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-webhook-cert" pod "cb490558-44fb-4262-a7ee-7318cca73d10" (UID: "cb490558-44fb-4262-a7ee-7318cca73d10"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:16:58.941402 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.941379 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb490558-44fb-4262-a7ee-7318cca73d10-kube-api-access-p7f26" pod "cb490558-44fb-4262-a7ee-7318cca73d10" (UID: "cb490558-44fb-4262-a7ee-7318cca73d10"). InnerVolumeSpecName "kube-api-access-p7f26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:16:58.958436 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.958385 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn"] Oct 30 05:16:58.959068 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.959048 4793 cpu_manager.go:414] "containerMap: removing container" podUID="cb490558-44fb-4262-a7ee-7318cca73d10" containerName="manager" Oct 30 05:16:58.959169 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.959072 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb490558-44fb-4262-a7ee-7318cca73d10" containerName="manager" Oct 30 05:16:58.959238 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.959225 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb490558-44fb-4262-a7ee-7318cca73d10" containerName="manager" Oct 30 05:16:58.962390 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.962370 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:58.982376 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:58.982345 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn"] Oct 30 05:16:59.040546 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.040509 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.040734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.040556 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.040734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.040637 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a556af0-394a-4c79-a3b8-971c207edf59-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.040877 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.040742 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjcgw\" (UniqueName: \"kubernetes.io/projected/4a556af0-394a-4c79-a3b8-971c207edf59-kube-api-access-zjcgw\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.040877 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.040790 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.040877 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.040833 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a556af0-394a-4c79-a3b8-971c207edf59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.041043 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.040876 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.041043 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.040930 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.041043 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.040970 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a556af0-394a-4c79-a3b8-971c207edf59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.041043 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.041023 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7f26\" (UniqueName: \"kubernetes.io/projected/cb490558-44fb-4262-a7ee-7318cca73d10-kube-api-access-p7f26\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:59.041254 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.041042 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:59.041254 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.041056 4793 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cb490558-44fb-4262-a7ee-7318cca73d10-webhook-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:16:59.141542 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.141475 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjcgw\" (UniqueName: \"kubernetes.io/projected/4a556af0-394a-4c79-a3b8-971c207edf59-kube-api-access-zjcgw\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.141542 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.141517 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.141774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.141551 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a556af0-394a-4c79-a3b8-971c207edf59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.141774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.141590 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.141774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.141638 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.141774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.141671 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a556af0-394a-4c79-a3b8-971c207edf59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.141774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.141713 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.141774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.141735 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.141774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.141773 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a556af0-394a-4c79-a3b8-971c207edf59-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.142178 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.141923 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.142178 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.141975 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.142288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.142200 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.142397 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.142374 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.142532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.142510 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a556af0-394a-4c79-a3b8-971c207edf59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.143987 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.143965 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a556af0-394a-4c79-a3b8-971c207edf59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.144088 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.144012 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a556af0-394a-4c79-a3b8-971c207edf59-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.163763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.163739 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a556af0-394a-4c79-a3b8-971c207edf59-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.163958 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.163941 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjcgw\" (UniqueName: \"kubernetes.io/projected/4a556af0-394a-4c79-a3b8-971c207edf59-kube-api-access-zjcgw\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mfjwn\" (UID: \"4a556af0-394a-4c79-a3b8-971c207edf59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.277032 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.277002 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:16:59.591883 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.591857 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn"] Oct 30 05:16:59.592880 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:16:59.592851 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a556af0_394a_4c79_a3b8_971c207edf59.slice/crio-b574897d01545be2b66d790e907acdb94ad7cfb2a6bd274b1206a29d746af11c WatchSource:0}: Error finding container b574897d01545be2b66d790e907acdb94ad7cfb2a6bd274b1206a29d746af11c: Status 404 returned error can't find the container with id b574897d01545be2b66d790e907acdb94ad7cfb2a6bd274b1206a29d746af11c Oct 30 05:16:59.595159 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.595128 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:16:59.595224 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.595191 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:16:59.595224 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.595220 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:16:59.758330 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.758281 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:16:59.884465 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.884363 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" event={"ID":"4a556af0-394a-4c79-a3b8-971c207edf59","Type":"ContainerStarted","Data":"cf45fb37c748adcdca213b9b304e9f745d4f97fdef5097d309ea03ab8031e424"} Oct 30 05:16:59.884622 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.884462 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" event={"ID":"4a556af0-394a-4c79-a3b8-971c207edf59","Type":"ContainerStarted","Data":"b574897d01545be2b66d790e907acdb94ad7cfb2a6bd274b1206a29d746af11c"} Oct 30 05:16:59.884622 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.884474 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f4559f84c-cb6r5" Oct 30 05:16:59.928681 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.928430 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" podStartSLOduration=1.928398608 podStartE2EDuration="1.928398608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:16:59.927433877 +0000 UTC m=+4743.213308646" watchObservedRunningTime="2026-10-30 05:16:59.928398608 +0000 UTC m=+4743.214273385" Oct 30 05:16:59.963873 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.963839 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f4559f84c-cb6r5"] Oct 30 05:16:59.969544 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:16:59.969513 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f4559f84c-cb6r5"] Oct 30 05:17:00.278446 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:00.277786 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:17:00.280141 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:00.280110 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mfjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 05:17:00.280284 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:00.280170 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" podUID="4a556af0-394a-4c79-a3b8-971c207edf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 05:17:00.845037 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:00.845003 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb490558-44fb-4262-a7ee-7318cca73d10" path="/var/lib/kubelet/pods/cb490558-44fb-4262-a7ee-7318cca73d10/volumes" Oct 30 05:17:01.106085 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:01.106002 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:17:01.278850 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:01.278810 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mfjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 05:17:01.279072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:01.278863 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" podUID="4a556af0-394a-4c79-a3b8-971c207edf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 05:17:02.278877 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:02.278842 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mfjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 05:17:02.279318 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:02.278898 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" podUID="4a556af0-394a-4c79-a3b8-971c207edf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 05:17:03.278801 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:03.278764 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mfjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 05:17:03.278982 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:03.278824 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" podUID="4a556af0-394a-4c79-a3b8-971c207edf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 05:17:04.278782 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:04.278746 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mfjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 05:17:04.278949 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:04.278801 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" podUID="4a556af0-394a-4c79-a3b8-971c207edf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 05:17:05.278716 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:05.278676 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mfjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 05:17:05.279238 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:05.278738 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" podUID="4a556af0-394a-4c79-a3b8-971c207edf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 05:17:06.277816 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:06.277776 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mfjwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 05:17:06.278018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:06.277844 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" podUID="4a556af0-394a-4c79-a3b8-971c207edf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 05:17:07.794198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:07.794166 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:17:07.794788 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:07.794754 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:17:07.795856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:07.795829 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" Oct 30 05:17:11.217964 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:17:11.217917 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:17:27.880444 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:27.880397 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg"] Oct 30 05:17:27.884177 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:27.884153 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:27.899910 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:27.899885 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:17:27.900147 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:27.900121 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:17:27.911451 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:27.911406 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:17:27.911611 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:27.911470 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:17:27.915191 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:27.915168 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg"] Oct 30 05:17:27.965500 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:27.965479 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-hfgdg\" (UID: \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:27.965615 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:27.965507 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkpc9\" (UniqueName: \"kubernetes.io/projected/dec27b38-191a-4560-a8d4-7d8f59d76e2e-kube-api-access-fkpc9\") pod \"maas-api-revoke-sub-f904bff9-hfgdg\" (UID: \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:27.965659 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:27.965640 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-hfgdg\" (UID: \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:28.067024 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:28.067000 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-hfgdg\" (UID: \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:28.067172 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:28.067088 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-hfgdg\" (UID: \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:28.067172 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:28.067118 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkpc9\" (UniqueName: \"kubernetes.io/projected/dec27b38-191a-4560-a8d4-7d8f59d76e2e-kube-api-access-fkpc9\") pod \"maas-api-revoke-sub-f904bff9-hfgdg\" (UID: \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:28.067704 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:28.067681 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-hfgdg\" (UID: \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:28.069405 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:28.069378 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-hfgdg\" (UID: \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:28.089408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:28.089388 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkpc9\" (UniqueName: \"kubernetes.io/projected/dec27b38-191a-4560-a8d4-7d8f59d76e2e-kube-api-access-fkpc9\") pod \"maas-api-revoke-sub-f904bff9-hfgdg\" (UID: \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:28.204529 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:28.204493 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:28.500279 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:28.500254 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg"] Oct 30 05:17:28.501935 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:17:28.501904 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec27b38_191a_4560_a8d4_7d8f59d76e2e.slice/crio-0e76813bab7e7b16167cf94834b7dd243f929c730e490ba205cbc033e18c40f9 WatchSource:0}: Error finding container 0e76813bab7e7b16167cf94834b7dd243f929c730e490ba205cbc033e18c40f9: Status 404 returned error can't find the container with id 0e76813bab7e7b16167cf94834b7dd243f929c730e490ba205cbc033e18c40f9 Oct 30 05:17:29.048404 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:29.048371 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" event={"ID":"dec27b38-191a-4560-a8d4-7d8f59d76e2e","Type":"ContainerStarted","Data":"873a2e3aca5925588af598308dcbbae9988927e5a1feb4c4cf2bfc882e5e7788"} Oct 30 05:17:29.048795 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:29.048443 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" event={"ID":"dec27b38-191a-4560-a8d4-7d8f59d76e2e","Type":"ContainerStarted","Data":"0e76813bab7e7b16167cf94834b7dd243f929c730e490ba205cbc033e18c40f9"} Oct 30 05:17:29.089495 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:29.089452 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" podStartSLOduration=2.089439686 podStartE2EDuration="2.089439686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:17:29.086357859 +0000 UTC m=+4772.372232633" watchObservedRunningTime="2026-10-30 05:17:29.089439686 +0000 UTC m=+4772.375314451" Oct 30 05:17:30.056146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:30.056108 4793 generic.go:356] "Generic (PLEG): container finished" podID="dec27b38-191a-4560-a8d4-7d8f59d76e2e" containerID="873a2e3aca5925588af598308dcbbae9988927e5a1feb4c4cf2bfc882e5e7788" exitCode=0 Oct 30 05:17:30.056675 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:30.056148 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" event={"ID":"dec27b38-191a-4560-a8d4-7d8f59d76e2e","Type":"ContainerDied","Data":"873a2e3aca5925588af598308dcbbae9988927e5a1feb4c4cf2bfc882e5e7788"} Oct 30 05:17:31.202722 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:31.202695 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:31.299706 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:31.299681 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec27b38-191a-4560-a8d4-7d8f59d76e2e-kube-api-access-fkpc9\" (UniqueName: \"kubernetes.io/projected/dec27b38-191a-4560-a8d4-7d8f59d76e2e-kube-api-access-fkpc9\") pod \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\" (UID: \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\") " Oct 30 05:17:31.299842 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:31.299715 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-openshift-ca\") pod \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\" (UID: \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\") " Oct 30 05:17:31.299842 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:31.299780 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-xks-ca\") pod \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\" (UID: \"dec27b38-191a-4560-a8d4-7d8f59d76e2e\") " Oct 30 05:17:31.300105 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:31.300085 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-openshift-ca" pod "dec27b38-191a-4560-a8d4-7d8f59d76e2e" (UID: "dec27b38-191a-4560-a8d4-7d8f59d76e2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:17:31.301873 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:31.301853 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec27b38-191a-4560-a8d4-7d8f59d76e2e-kube-api-access-fkpc9" pod "dec27b38-191a-4560-a8d4-7d8f59d76e2e" (UID: "dec27b38-191a-4560-a8d4-7d8f59d76e2e"). InnerVolumeSpecName "kube-api-access-fkpc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:17:31.301940 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:31.301917 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-xks-ca" pod "dec27b38-191a-4560-a8d4-7d8f59d76e2e" (UID: "dec27b38-191a-4560-a8d4-7d8f59d76e2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:17:31.401561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:31.401504 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkpc9\" (UniqueName: \"kubernetes.io/projected/dec27b38-191a-4560-a8d4-7d8f59d76e2e-kube-api-access-fkpc9\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:17:31.401561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:31.401527 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:17:31.401561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:31.401537 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dec27b38-191a-4560-a8d4-7d8f59d76e2e-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:17:32.067299 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:32.067273 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" Oct 30 05:17:32.067501 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:32.067321 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg" event={"ID":"dec27b38-191a-4560-a8d4-7d8f59d76e2e","Type":"ContainerDied","Data":"0e76813bab7e7b16167cf94834b7dd243f929c730e490ba205cbc033e18c40f9"} Oct 30 05:17:32.067501 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:32.067375 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e76813bab7e7b16167cf94834b7dd243f929c730e490ba205cbc033e18c40f9" Oct 30 05:17:41.226401 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:17:41.226360 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:17:52.598020 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.597987 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx"] Oct 30 05:17:52.598458 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.598395 4793 cpu_manager.go:414] "containerMap: removing container" podUID="dec27b38-191a-4560-a8d4-7d8f59d76e2e" containerName="revoke-keys" Oct 30 05:17:52.598458 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.598410 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec27b38-191a-4560-a8d4-7d8f59d76e2e" containerName="revoke-keys" Oct 30 05:17:52.598579 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.598529 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec27b38-191a-4560-a8d4-7d8f59d76e2e" containerName="revoke-keys" Oct 30 05:17:52.601730 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.601712 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:52.630089 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.630058 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:17:52.630239 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.630180 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:17:52.630239 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.630228 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:17:52.630326 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.630309 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:17:52.632940 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.632916 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx"] Oct 30 05:17:52.737613 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.737585 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xrdfx\" (UID: \"73a49595-22f0-42d5-9e92-ba8920518f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:52.737743 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.737621 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xrdfx\" (UID: \"73a49595-22f0-42d5-9e92-ba8920518f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:52.737743 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.737707 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xj4h\" (UniqueName: \"kubernetes.io/projected/73a49595-22f0-42d5-9e92-ba8920518f83-kube-api-access-6xj4h\") pod \"maas-api-revoke-sub-f03629db-xrdfx\" (UID: \"73a49595-22f0-42d5-9e92-ba8920518f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:52.838258 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.838226 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xrdfx\" (UID: \"73a49595-22f0-42d5-9e92-ba8920518f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:52.838445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.838267 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xrdfx\" (UID: \"73a49595-22f0-42d5-9e92-ba8920518f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:52.838445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.838321 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xj4h\" (UniqueName: \"kubernetes.io/projected/73a49595-22f0-42d5-9e92-ba8920518f83-kube-api-access-6xj4h\") pod \"maas-api-revoke-sub-f03629db-xrdfx\" (UID: \"73a49595-22f0-42d5-9e92-ba8920518f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:52.838986 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.838961 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-xrdfx\" (UID: \"73a49595-22f0-42d5-9e92-ba8920518f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:52.840829 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.840806 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-xrdfx\" (UID: \"73a49595-22f0-42d5-9e92-ba8920518f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:52.861753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.861702 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xj4h\" (UniqueName: \"kubernetes.io/projected/73a49595-22f0-42d5-9e92-ba8920518f83-kube-api-access-6xj4h\") pod \"maas-api-revoke-sub-f03629db-xrdfx\" (UID: \"73a49595-22f0-42d5-9e92-ba8920518f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:52.915753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:52.915723 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:53.212683 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:53.212654 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx"] Oct 30 05:17:53.214574 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:17:53.214545 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a49595_22f0_42d5_9e92_ba8920518f83.slice/crio-ce980d47419b4c4aa5658a2950e1bab71df2d3b611aaed58889b01ada60345a8 WatchSource:0}: Error finding container ce980d47419b4c4aa5658a2950e1bab71df2d3b611aaed58889b01ada60345a8: Status 404 returned error can't find the container with id ce980d47419b4c4aa5658a2950e1bab71df2d3b611aaed58889b01ada60345a8 Oct 30 05:17:54.227357 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:54.227266 4793 generic.go:356] "Generic (PLEG): container finished" podID="73a49595-22f0-42d5-9e92-ba8920518f83" containerID="1bc57d42fec6e0b80869f49feae702a2ece7f0622a52007b57ce4bffadb8d74a" exitCode=0 Oct 30 05:17:54.227357 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:54.227311 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" event={"ID":"73a49595-22f0-42d5-9e92-ba8920518f83","Type":"ContainerDied","Data":"1bc57d42fec6e0b80869f49feae702a2ece7f0622a52007b57ce4bffadb8d74a"} Oct 30 05:17:54.227357 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:54.227356 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" event={"ID":"73a49595-22f0-42d5-9e92-ba8920518f83","Type":"ContainerStarted","Data":"ce980d47419b4c4aa5658a2950e1bab71df2d3b611aaed58889b01ada60345a8"} Oct 30 05:17:55.376430 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:55.376393 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:55.460953 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:55.460928 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-openshift-ca\") pod \"73a49595-22f0-42d5-9e92-ba8920518f83\" (UID: \"73a49595-22f0-42d5-9e92-ba8920518f83\") " Oct 30 05:17:55.461109 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:55.460969 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a49595-22f0-42d5-9e92-ba8920518f83-kube-api-access-6xj4h\" (UniqueName: \"kubernetes.io/projected/73a49595-22f0-42d5-9e92-ba8920518f83-kube-api-access-6xj4h\") pod \"73a49595-22f0-42d5-9e92-ba8920518f83\" (UID: \"73a49595-22f0-42d5-9e92-ba8920518f83\") " Oct 30 05:17:55.461109 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:55.461031 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-xks-ca\") pod \"73a49595-22f0-42d5-9e92-ba8920518f83\" (UID: \"73a49595-22f0-42d5-9e92-ba8920518f83\") " Oct 30 05:17:55.461299 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:55.461276 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-openshift-ca" pod "73a49595-22f0-42d5-9e92-ba8920518f83" (UID: "73a49595-22f0-42d5-9e92-ba8920518f83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:17:55.461445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:55.461408 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:17:55.463098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:55.463069 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-xks-ca" pod "73a49595-22f0-42d5-9e92-ba8920518f83" (UID: "73a49595-22f0-42d5-9e92-ba8920518f83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:17:55.463219 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:55.463201 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a49595-22f0-42d5-9e92-ba8920518f83-kube-api-access-6xj4h" pod "73a49595-22f0-42d5-9e92-ba8920518f83" (UID: "73a49595-22f0-42d5-9e92-ba8920518f83"). InnerVolumeSpecName "kube-api-access-6xj4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:17:55.562010 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:55.561941 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xj4h\" (UniqueName: \"kubernetes.io/projected/73a49595-22f0-42d5-9e92-ba8920518f83-kube-api-access-6xj4h\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:17:55.562010 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:55.561965 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a49595-22f0-42d5-9e92-ba8920518f83-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:17:56.239284 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:56.239249 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" Oct 30 05:17:56.239500 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:56.239276 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx" event={"ID":"73a49595-22f0-42d5-9e92-ba8920518f83","Type":"ContainerDied","Data":"ce980d47419b4c4aa5658a2950e1bab71df2d3b611aaed58889b01ada60345a8"} Oct 30 05:17:56.239500 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:56.239350 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce980d47419b4c4aa5658a2950e1bab71df2d3b611aaed58889b01ada60345a8" Oct 30 05:17:56.549205 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:56.549117 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:17:58.394280 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:58.394238 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:17:58.841931 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:58.841899 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:17:59.070158 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:59.070116 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:17:59.873489 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:17:59.873453 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:18:05.472746 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:05.472712 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:18:05.710455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:05.710317 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:18:08.754525 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.754483 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z"] Oct 30 05:18:08.754959 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.754942 4793 cpu_manager.go:414] "containerMap: removing container" podUID="73a49595-22f0-42d5-9e92-ba8920518f83" containerName="revoke-keys" Oct 30 05:18:08.755008 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.754962 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a49595-22f0-42d5-9e92-ba8920518f83" containerName="revoke-keys" Oct 30 05:18:08.755045 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.755032 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a49595-22f0-42d5-9e92-ba8920518f83" containerName="revoke-keys" Oct 30 05:18:08.759012 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.758990 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:08.776452 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.776409 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z"] Oct 30 05:18:08.785397 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.785376 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:18:08.785553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.785410 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:18:08.785553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.785440 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:18:08.785553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.785514 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:18:08.932331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.932303 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hng5z\" (UID: \"b34c9d7e-8403-40d6-a187-176c465b85a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:08.932539 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.932433 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hng5z\" (UID: \"b34c9d7e-8403-40d6-a187-176c465b85a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:08.932539 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:08.932511 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qtpx\" (UniqueName: \"kubernetes.io/projected/b34c9d7e-8403-40d6-a187-176c465b85a3-kube-api-access-9qtpx\") pod \"maas-api-revoke-sub-0165d7d2-hng5z\" (UID: \"b34c9d7e-8403-40d6-a187-176c465b85a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:09.033284 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:09.033205 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hng5z\" (UID: \"b34c9d7e-8403-40d6-a187-176c465b85a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:09.033284 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:09.033244 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qtpx\" (UniqueName: \"kubernetes.io/projected/b34c9d7e-8403-40d6-a187-176c465b85a3-kube-api-access-9qtpx\") pod \"maas-api-revoke-sub-0165d7d2-hng5z\" (UID: \"b34c9d7e-8403-40d6-a187-176c465b85a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:09.033534 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:09.033297 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hng5z\" (UID: \"b34c9d7e-8403-40d6-a187-176c465b85a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:09.033877 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:09.033857 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-hng5z\" (UID: \"b34c9d7e-8403-40d6-a187-176c465b85a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:09.035480 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:09.035454 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-hng5z\" (UID: \"b34c9d7e-8403-40d6-a187-176c465b85a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:09.055935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:09.055909 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qtpx\" (UniqueName: \"kubernetes.io/projected/b34c9d7e-8403-40d6-a187-176c465b85a3-kube-api-access-9qtpx\") pod \"maas-api-revoke-sub-0165d7d2-hng5z\" (UID: \"b34c9d7e-8403-40d6-a187-176c465b85a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:09.072751 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:09.072728 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:09.367611 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:09.367588 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z"] Oct 30 05:18:09.369284 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:18:09.369254 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34c9d7e_8403_40d6_a187_176c465b85a3.slice/crio-5be7b2acb379d25ca8ba4e69ca5aa204a3ad72698fef071acea9df3cd8909fb5 WatchSource:0}: Error finding container 5be7b2acb379d25ca8ba4e69ca5aa204a3ad72698fef071acea9df3cd8909fb5: Status 404 returned error can't find the container with id 5be7b2acb379d25ca8ba4e69ca5aa204a3ad72698fef071acea9df3cd8909fb5 Oct 30 05:18:10.324793 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:10.324760 4793 generic.go:356] "Generic (PLEG): container finished" podID="b34c9d7e-8403-40d6-a187-176c465b85a3" containerID="5304ba820fc55768b75119e093d8e086cd0d193dae0bedfbc71644bdb599451f" exitCode=0 Oct 30 05:18:10.325181 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:10.324847 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" event={"ID":"b34c9d7e-8403-40d6-a187-176c465b85a3","Type":"ContainerDied","Data":"5304ba820fc55768b75119e093d8e086cd0d193dae0bedfbc71644bdb599451f"} Oct 30 05:18:10.325181 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:10.324897 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" event={"ID":"b34c9d7e-8403-40d6-a187-176c465b85a3","Type":"ContainerStarted","Data":"5be7b2acb379d25ca8ba4e69ca5aa204a3ad72698fef071acea9df3cd8909fb5"} Oct 30 05:18:11.218876 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:18:11.218846 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:18:11.460465 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:11.460438 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:11.556546 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:11.556488 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-openshift-ca\") pod \"b34c9d7e-8403-40d6-a187-176c465b85a3\" (UID: \"b34c9d7e-8403-40d6-a187-176c465b85a3\") " Oct 30 05:18:11.556690 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:11.556548 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-xks-ca\") pod \"b34c9d7e-8403-40d6-a187-176c465b85a3\" (UID: \"b34c9d7e-8403-40d6-a187-176c465b85a3\") " Oct 30 05:18:11.556690 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:11.556603 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34c9d7e-8403-40d6-a187-176c465b85a3-kube-api-access-9qtpx\" (UniqueName: \"kubernetes.io/projected/b34c9d7e-8403-40d6-a187-176c465b85a3-kube-api-access-9qtpx\") pod \"b34c9d7e-8403-40d6-a187-176c465b85a3\" (UID: \"b34c9d7e-8403-40d6-a187-176c465b85a3\") " Oct 30 05:18:11.556908 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:11.556880 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-openshift-ca" pod "b34c9d7e-8403-40d6-a187-176c465b85a3" (UID: "b34c9d7e-8403-40d6-a187-176c465b85a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:11.558698 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:11.558676 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-xks-ca" pod "b34c9d7e-8403-40d6-a187-176c465b85a3" (UID: "b34c9d7e-8403-40d6-a187-176c465b85a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:18:11.558784 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:11.558745 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34c9d7e-8403-40d6-a187-176c465b85a3-kube-api-access-9qtpx" pod "b34c9d7e-8403-40d6-a187-176c465b85a3" (UID: "b34c9d7e-8403-40d6-a187-176c465b85a3"). InnerVolumeSpecName "kube-api-access-9qtpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:11.657963 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:11.657938 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:18:11.658055 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:11.657965 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34c9d7e-8403-40d6-a187-176c465b85a3-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:18:11.658055 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:11.657975 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qtpx\" (UniqueName: \"kubernetes.io/projected/b34c9d7e-8403-40d6-a187-176c465b85a3-kube-api-access-9qtpx\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:18:12.337793 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:12.337756 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" event={"ID":"b34c9d7e-8403-40d6-a187-176c465b85a3","Type":"ContainerDied","Data":"5be7b2acb379d25ca8ba4e69ca5aa204a3ad72698fef071acea9df3cd8909fb5"} Oct 30 05:18:12.337966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:12.337798 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z" Oct 30 05:18:12.337966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:12.337837 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5be7b2acb379d25ca8ba4e69ca5aa204a3ad72698fef071acea9df3cd8909fb5" Oct 30 05:18:19.711575 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.711536 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d"] Oct 30 05:18:19.712070 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.712054 4793 cpu_manager.go:414] "containerMap: removing container" podUID="b34c9d7e-8403-40d6-a187-176c465b85a3" containerName="revoke-keys" Oct 30 05:18:19.712120 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.712072 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34c9d7e-8403-40d6-a187-176c465b85a3" containerName="revoke-keys" Oct 30 05:18:19.712157 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.712145 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34c9d7e-8403-40d6-a187-176c465b85a3" containerName="revoke-keys" Oct 30 05:18:19.715311 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.715292 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:19.723640 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.723619 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-78n5d\" (UID: \"33d1e89c-683f-4dcc-b71c-240b846494cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:19.723721 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.723666 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-78n5d\" (UID: \"33d1e89c-683f-4dcc-b71c-240b846494cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:19.723808 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.723787 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22sv7\" (UniqueName: \"kubernetes.io/projected/33d1e89c-683f-4dcc-b71c-240b846494cc-kube-api-access-22sv7\") pod \"maas-api-revoke-sub-6b19338f-78n5d\" (UID: \"33d1e89c-683f-4dcc-b71c-240b846494cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:19.743051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.743009 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:18:19.743177 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.743054 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:18:19.743265 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.743216 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:18:19.743341 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.743314 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:18:19.747081 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.747061 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d"] Oct 30 05:18:19.824593 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.824558 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-78n5d\" (UID: \"33d1e89c-683f-4dcc-b71c-240b846494cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:19.824763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.824625 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-78n5d\" (UID: \"33d1e89c-683f-4dcc-b71c-240b846494cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:19.824763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.824712 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22sv7\" (UniqueName: \"kubernetes.io/projected/33d1e89c-683f-4dcc-b71c-240b846494cc-kube-api-access-22sv7\") pod \"maas-api-revoke-sub-6b19338f-78n5d\" (UID: \"33d1e89c-683f-4dcc-b71c-240b846494cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:19.825314 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.825296 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-78n5d\" (UID: \"33d1e89c-683f-4dcc-b71c-240b846494cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:19.826970 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.826951 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-78n5d\" (UID: \"33d1e89c-683f-4dcc-b71c-240b846494cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:19.847533 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:19.847506 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22sv7\" (UniqueName: \"kubernetes.io/projected/33d1e89c-683f-4dcc-b71c-240b846494cc-kube-api-access-22sv7\") pod \"maas-api-revoke-sub-6b19338f-78n5d\" (UID: \"33d1e89c-683f-4dcc-b71c-240b846494cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:20.028230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:20.028153 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:20.325995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:20.325903 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d"] Oct 30 05:18:20.331913 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:18:20.331884 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d1e89c_683f_4dcc_b71c_240b846494cc.slice/crio-e4c803220bf60a0fba9860a5ec9651c787afe67bca7039dc18a7a4012eb2203a WatchSource:0}: Error finding container e4c803220bf60a0fba9860a5ec9651c787afe67bca7039dc18a7a4012eb2203a: Status 404 returned error can't find the container with id e4c803220bf60a0fba9860a5ec9651c787afe67bca7039dc18a7a4012eb2203a Oct 30 05:18:20.383976 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:20.383945 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" event={"ID":"33d1e89c-683f-4dcc-b71c-240b846494cc","Type":"ContainerStarted","Data":"e4c803220bf60a0fba9860a5ec9651c787afe67bca7039dc18a7a4012eb2203a"} Oct 30 05:18:21.391252 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:21.391222 4793 generic.go:356] "Generic (PLEG): container finished" podID="33d1e89c-683f-4dcc-b71c-240b846494cc" containerID="01c8812f5840befc9ba62b083ff930b6c3843ec3f4981614d734de9dc0abf2e8" exitCode=0 Oct 30 05:18:21.391639 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:21.391305 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" event={"ID":"33d1e89c-683f-4dcc-b71c-240b846494cc","Type":"ContainerDied","Data":"01c8812f5840befc9ba62b083ff930b6c3843ec3f4981614d734de9dc0abf2e8"} Oct 30 05:18:22.528164 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:22.528131 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:22.547332 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:22.547307 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-xks-ca\") pod \"33d1e89c-683f-4dcc-b71c-240b846494cc\" (UID: \"33d1e89c-683f-4dcc-b71c-240b846494cc\") " Oct 30 05:18:22.547468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:22.547362 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-openshift-ca\") pod \"33d1e89c-683f-4dcc-b71c-240b846494cc\" (UID: \"33d1e89c-683f-4dcc-b71c-240b846494cc\") " Oct 30 05:18:22.547527 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:22.547474 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d1e89c-683f-4dcc-b71c-240b846494cc-kube-api-access-22sv7\" (UniqueName: \"kubernetes.io/projected/33d1e89c-683f-4dcc-b71c-240b846494cc-kube-api-access-22sv7\") pod \"33d1e89c-683f-4dcc-b71c-240b846494cc\" (UID: \"33d1e89c-683f-4dcc-b71c-240b846494cc\") " Oct 30 05:18:22.547692 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:22.547669 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-openshift-ca" pod "33d1e89c-683f-4dcc-b71c-240b846494cc" (UID: "33d1e89c-683f-4dcc-b71c-240b846494cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:18:22.547865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:22.547850 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:18:22.549288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:22.549266 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-xks-ca" pod "33d1e89c-683f-4dcc-b71c-240b846494cc" (UID: "33d1e89c-683f-4dcc-b71c-240b846494cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:18:22.549451 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:22.549430 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d1e89c-683f-4dcc-b71c-240b846494cc-kube-api-access-22sv7" pod "33d1e89c-683f-4dcc-b71c-240b846494cc" (UID: "33d1e89c-683f-4dcc-b71c-240b846494cc"). InnerVolumeSpecName "kube-api-access-22sv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:18:22.648643 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:22.648581 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22sv7\" (UniqueName: \"kubernetes.io/projected/33d1e89c-683f-4dcc-b71c-240b846494cc-kube-api-access-22sv7\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:18:22.648643 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:22.648604 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d1e89c-683f-4dcc-b71c-240b846494cc-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:18:23.402655 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:23.402622 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" Oct 30 05:18:23.402838 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:23.402649 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d" event={"ID":"33d1e89c-683f-4dcc-b71c-240b846494cc","Type":"ContainerDied","Data":"e4c803220bf60a0fba9860a5ec9651c787afe67bca7039dc18a7a4012eb2203a"} Oct 30 05:18:23.402838 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:23.402703 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4c803220bf60a0fba9860a5ec9651c787afe67bca7039dc18a7a4012eb2203a" Oct 30 05:18:32.536198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.536157 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:18:32.768324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.768235 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:18:32.811399 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.811366 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s"] Oct 30 05:18:32.812200 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.812173 4793 cpu_manager.go:414] "containerMap: removing container" podUID="33d1e89c-683f-4dcc-b71c-240b846494cc" containerName="revoke-keys" Oct 30 05:18:32.812357 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.812343 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d1e89c-683f-4dcc-b71c-240b846494cc" containerName="revoke-keys" Oct 30 05:18:32.818714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.818695 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d1e89c-683f-4dcc-b71c-240b846494cc" containerName="revoke-keys" Oct 30 05:18:32.823960 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.822892 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:32.832435 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.832393 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s"] Oct 30 05:18:32.847446 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.844492 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-14305626-kserve-self-signed-certs\"" Oct 30 05:18:32.949505 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.949465 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-tmp-dir\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:32.949698 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.949521 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53146743-4864-492b-b5f4-238ec87050b7-tls-certs\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:32.949698 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.949615 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrh2w\" (UniqueName: \"kubernetes.io/projected/53146743-4864-492b-b5f4-238ec87050b7-kube-api-access-rrh2w\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:32.949698 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.949651 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-dshm\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:32.949899 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.949745 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-home\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:32.949899 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:32.949769 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-model-cache\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.046055 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.045957 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:18:33.053212 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.053181 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-tmp-dir\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.053371 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.053217 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53146743-4864-492b-b5f4-238ec87050b7-tls-certs\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.053545 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.053520 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrh2w\" (UniqueName: \"kubernetes.io/projected/53146743-4864-492b-b5f4-238ec87050b7-kube-api-access-rrh2w\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.053985 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.053961 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-dshm\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.054206 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.054186 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-home\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.054330 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.054314 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-model-cache\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.054798 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.054775 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-model-cache\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.054938 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.053733 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-tmp-dir\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.055585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.055542 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-home\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.072629 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.072582 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-dshm\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.086921 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.078905 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53146743-4864-492b-b5f4-238ec87050b7-tls-certs\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.105444 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.096108 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrh2w\" (UniqueName: \"kubernetes.io/projected/53146743-4864-492b-b5f4-238ec87050b7-kube-api-access-rrh2w\") pod \"e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.209471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.207958 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz"] Oct 30 05:18:33.235271 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.235235 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:18:33.239874 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.239835 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz"] Oct 30 05:18:33.257702 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.250336 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:33.257702 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.257667 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-14305626-epp-sa-dockercfg-hs95z\"" Oct 30 05:18:33.270241 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.270207 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d546841-6271-4765-98c7-f4f4d628f02e-tls-certs\") pod \"e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz\" (UID: \"6d546841-6271-4765-98c7-f4f4d628f02e\") " pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:18:33.270458 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.270404 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpq8\" (UniqueName: \"kubernetes.io/projected/6d546841-6271-4765-98c7-f4f4d628f02e-kube-api-access-wmpq8\") pod \"e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz\" (UID: \"6d546841-6271-4765-98c7-f4f4d628f02e\") " pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:18:33.371319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.371287 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpq8\" (UniqueName: \"kubernetes.io/projected/6d546841-6271-4765-98c7-f4f4d628f02e-kube-api-access-wmpq8\") pod \"e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz\" (UID: \"6d546841-6271-4765-98c7-f4f4d628f02e\") " pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:18:33.371506 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.371359 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d546841-6271-4765-98c7-f4f4d628f02e-tls-certs\") pod \"e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz\" (UID: \"6d546841-6271-4765-98c7-f4f4d628f02e\") " pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:18:33.373759 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.373737 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d546841-6271-4765-98c7-f4f4d628f02e-tls-certs\") pod \"e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz\" (UID: \"6d546841-6271-4765-98c7-f4f4d628f02e\") " pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:18:33.409575 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.409552 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpq8\" (UniqueName: \"kubernetes.io/projected/6d546841-6271-4765-98c7-f4f4d628f02e-kube-api-access-wmpq8\") pod \"e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz\" (UID: \"6d546841-6271-4765-98c7-f4f4d628f02e\") " pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:18:33.482135 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.482084 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:18:33.568858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.568836 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s"] Oct 30 05:18:33.570986 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:18:33.570957 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53146743_4864_492b_b5f4_238ec87050b7.slice/crio-7a71ead24e916ca24b42c5774bf5c60389f06e1bff8ae8f46f4ba7facfadc510 WatchSource:0}: Error finding container 7a71ead24e916ca24b42c5774bf5c60389f06e1bff8ae8f46f4ba7facfadc510: Status 404 returned error can't find the container with id 7a71ead24e916ca24b42c5774bf5c60389f06e1bff8ae8f46f4ba7facfadc510 Oct 30 05:18:33.589065 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.589045 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:18:33.892349 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:33.892325 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz"] Oct 30 05:18:33.894185 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:18:33.894160 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d546841_6271_4765_98c7_f4f4d628f02e.slice/crio-9e61e0d4aadde48af7a42cd61c3417abbffa552384491f6fb894781c0b39e071 WatchSource:0}: Error finding container 9e61e0d4aadde48af7a42cd61c3417abbffa552384491f6fb894781c0b39e071: Status 404 returned error can't find the container with id 9e61e0d4aadde48af7a42cd61c3417abbffa552384491f6fb894781c0b39e071 Oct 30 05:18:34.480766 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:34.480726 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" event={"ID":"53146743-4864-492b-b5f4-238ec87050b7","Type":"ContainerStarted","Data":"ef08138d32edf89fba8751722691cc83abe33ba21470ca5de08bba2142746838"} Oct 30 05:18:34.480945 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:34.480780 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" event={"ID":"53146743-4864-492b-b5f4-238ec87050b7","Type":"ContainerStarted","Data":"7a71ead24e916ca24b42c5774bf5c60389f06e1bff8ae8f46f4ba7facfadc510"} Oct 30 05:18:34.482278 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:34.482248 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" event={"ID":"6d546841-6271-4765-98c7-f4f4d628f02e","Type":"ContainerStarted","Data":"9e61e0d4aadde48af7a42cd61c3417abbffa552384491f6fb894781c0b39e071"} Oct 30 05:18:34.525025 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:34.524970 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" podStartSLOduration=2.524951401 podStartE2EDuration="2.524951401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:34.52147703 +0000 UTC m=+4837.807351826" watchObservedRunningTime="2026-10-30 05:18:34.524951401 +0000 UTC m=+4837.810826177" Oct 30 05:18:35.250936 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:35.250901 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:35.899025 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:35.898939 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:18:36.499504 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:36.499357 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" event={"ID":"6d546841-6271-4765-98c7-f4f4d628f02e","Type":"ContainerStarted","Data":"c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace"} Oct 30 05:18:36.500571 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:36.500332 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:18:36.542697 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:36.542638 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" podStartSLOduration=1.9330940349999999 podStartE2EDuration="3.542621602s" totalImagesPullingTime="1.609527567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:18:36.539902767 +0000 UTC m=+4839.825777570" watchObservedRunningTime="2026-10-30 05:18:36.542621602 +0000 UTC m=+4839.828496381" Oct 30 05:18:41.218096 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:18:41.218060 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:18:43.251278 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:43.251242 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:43.264361 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:43.264335 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:18:43.562541 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:18:43.562469 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:19:00.027180 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:00.027144 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:06.955312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:06.955265 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:07.170477 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:07.170444 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:07.519374 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:07.519346 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:19:11.217909 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:19:11.217873 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:19:34.738995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:34.738955 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s"] Oct 30 05:19:34.745754 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:34.745722 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz"] Oct 30 05:19:34.853569 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:34.853522 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:35.086061 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:35.085966 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:37.518946 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:37.518910 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" podUID="6d546841-6271-4765-98c7-f4f4d628f02e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:19:41.219481 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:19:41.219376 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:19:46.180893 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:46.180856 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:47.525745 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:47.525700 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" podUID="6d546841-6271-4765-98c7-f4f4d628f02e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:19:49.766412 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:49.766352 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" podUID="53146743-4864-492b-b5f4-238ec87050b7" containerName="main" containerID="cri-o://ef08138d32edf89fba8751722691cc83abe33ba21470ca5de08bba2142746838" gracePeriod=45 Oct 30 05:19:49.766412 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:49.766351 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" podUID="6d546841-6271-4765-98c7-f4f4d628f02e" containerName="main" containerID="cri-o://c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace" gracePeriod=45 Oct 30 05:19:49.822146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:49.822106 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:49.973849 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:49.973819 4793 generic.go:356] "Generic (PLEG): container finished" podID="53146743-4864-492b-b5f4-238ec87050b7" containerID="ef08138d32edf89fba8751722691cc83abe33ba21470ca5de08bba2142746838" exitCode=0 Oct 30 05:19:49.974003 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:49.973896 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" event={"ID":"53146743-4864-492b-b5f4-238ec87050b7","Type":"ContainerDied","Data":"ef08138d32edf89fba8751722691cc83abe33ba21470ca5de08bba2142746838"} Oct 30 05:19:50.024258 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.024206 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:19:50.079577 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.079548 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-tmp-dir\") pod \"53146743-4864-492b-b5f4-238ec87050b7\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " Oct 30 05:19:50.079763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.079613 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-home\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-home\") pod \"53146743-4864-492b-b5f4-238ec87050b7\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " Oct 30 05:19:50.079763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.079683 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-dshm\") pod \"53146743-4864-492b-b5f4-238ec87050b7\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " Oct 30 05:19:50.079763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.079739 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53146743-4864-492b-b5f4-238ec87050b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/53146743-4864-492b-b5f4-238ec87050b7-tls-certs\") pod \"53146743-4864-492b-b5f4-238ec87050b7\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " Oct 30 05:19:50.079960 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.079802 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53146743-4864-492b-b5f4-238ec87050b7-kube-api-access-rrh2w\" (UniqueName: \"kubernetes.io/projected/53146743-4864-492b-b5f4-238ec87050b7-kube-api-access-rrh2w\") pod \"53146743-4864-492b-b5f4-238ec87050b7\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " Oct 30 05:19:50.079960 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.079832 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-tmp-dir" pod "53146743-4864-492b-b5f4-238ec87050b7" (UID: "53146743-4864-492b-b5f4-238ec87050b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:50.079960 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.079914 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-home" pod "53146743-4864-492b-b5f4-238ec87050b7" (UID: "53146743-4864-492b-b5f4-238ec87050b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:50.080141 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.079964 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-model-cache\") pod \"53146743-4864-492b-b5f4-238ec87050b7\" (UID: \"53146743-4864-492b-b5f4-238ec87050b7\") " Oct 30 05:19:50.080311 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.080286 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-model-cache" pod "53146743-4864-492b-b5f4-238ec87050b7" (UID: "53146743-4864-492b-b5f4-238ec87050b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:50.080508 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.080404 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:19:50.080618 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.080524 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:19:50.082543 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.082516 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-dshm" pod "53146743-4864-492b-b5f4-238ec87050b7" (UID: "53146743-4864-492b-b5f4-238ec87050b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:19:50.082543 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.082529 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53146743-4864-492b-b5f4-238ec87050b7-tls-certs" pod "53146743-4864-492b-b5f4-238ec87050b7" (UID: "53146743-4864-492b-b5f4-238ec87050b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:19:50.082801 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.082784 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53146743-4864-492b-b5f4-238ec87050b7-kube-api-access-rrh2w" pod "53146743-4864-492b-b5f4-238ec87050b7" (UID: "53146743-4864-492b-b5f4-238ec87050b7"). InnerVolumeSpecName "kube-api-access-rrh2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:19:50.181945 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.181898 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:19:50.181945 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.181944 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53146743-4864-492b-b5f4-238ec87050b7-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:19:50.182384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.181962 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrh2w\" (UniqueName: \"kubernetes.io/projected/53146743-4864-492b-b5f4-238ec87050b7-kube-api-access-rrh2w\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:19:50.182384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.181971 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53146743-4864-492b-b5f4-238ec87050b7-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:19:50.339052 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.338951 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:50.553403 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.553366 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:50.980317 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.980287 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" event={"ID":"53146743-4864-492b-b5f4-238ec87050b7","Type":"ContainerDied","Data":"7a71ead24e916ca24b42c5774bf5c60389f06e1bff8ae8f46f4ba7facfadc510"} Oct 30 05:19:50.980317 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.980314 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s" Oct 30 05:19:50.980790 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:50.980339 4793 scope.go:122] "RemoveContainer" containerID="ef08138d32edf89fba8751722691cc83abe33ba21470ca5de08bba2142746838" Oct 30 05:19:51.031212 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:51.031188 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s"] Oct 30 05:19:51.042999 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:51.042970 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-14305626-kserve-57b87f7fcd-7l77s"] Oct 30 05:19:52.844839 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:52.844802 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53146743-4864-492b-b5f4-238ec87050b7" path="/var/lib/kubelet/pods/53146743-4864-492b-b5f4-238ec87050b7/volumes" Oct 30 05:19:54.290651 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:54.290606 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:54.533772 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:54.533730 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:56.141939 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:56.141897 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:56.400012 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:56.399940 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:57.519360 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:57.519326 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" podUID="6d546841-6271-4765-98c7-f4f4d628f02e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:19:57.519782 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:57.519450 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:19:57.989387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:57.989355 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:19:59.864706 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:19:59.864670 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:00.166517 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:00.166395 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:01.778026 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:01.777986 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:05.347555 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:05.347520 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:05.706658 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:05.706628 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:07.519446 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:07.519379 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" podUID="6d546841-6271-4765-98c7-f4f4d628f02e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:20:07.680374 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:07.680334 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:09.284073 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:09.284028 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:09.601101 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:09.601011 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:09.959939 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:09.959898 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:10.079660 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:10.079623 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr"] Oct 30 05:20:10.105840 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:10.105804 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8dea38ec-5lpvr"] Oct 30 05:20:10.313008 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:10.312936 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:10.844152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:10.844119 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29fd98c6-e26f-4d08-a688-0f3b31404ebd" path="/var/lib/kubelet/pods/29fd98c6-e26f-4d08-a688-0f3b31404ebd/volumes" Oct 30 05:20:11.218785 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:20:11.218750 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:20:11.748539 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:11.748506 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:11.975312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:11.975278 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:13.799825 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:13.799789 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:14.570721 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:14.570682 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:15.822383 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:15.822345 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:17.519969 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:17.519930 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" podUID="6d546841-6271-4765-98c7-f4f4d628f02e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:20:17.754735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:17.754698 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:18.060016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:18.059954 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg"] Oct 30 05:20:18.071166 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:18.071130 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb533423-chhrg"] Oct 30 05:20:18.846114 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:18.846079 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b2e7fe-1ed9-471e-8169-3ce58d8c7c79" path="/var/lib/kubelet/pods/99b2e7fe-1ed9-471e-8169-3ce58d8c7c79/volumes" Oct 30 05:20:19.649551 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:19.649513 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:20.033720 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.033696 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:20:20.062360 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.062335 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d546841-6271-4765-98c7-f4f4d628f02e-kube-api-access-wmpq8\" (UniqueName: \"kubernetes.io/projected/6d546841-6271-4765-98c7-f4f4d628f02e-kube-api-access-wmpq8\") pod \"6d546841-6271-4765-98c7-f4f4d628f02e\" (UID: \"6d546841-6271-4765-98c7-f4f4d628f02e\") " Oct 30 05:20:20.062504 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.062494 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d546841-6271-4765-98c7-f4f4d628f02e-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d546841-6271-4765-98c7-f4f4d628f02e-tls-certs\") pod \"6d546841-6271-4765-98c7-f4f4d628f02e\" (UID: \"6d546841-6271-4765-98c7-f4f4d628f02e\") " Oct 30 05:20:20.064329 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.064309 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d546841-6271-4765-98c7-f4f4d628f02e-kube-api-access-wmpq8" pod "6d546841-6271-4765-98c7-f4f4d628f02e" (UID: "6d546841-6271-4765-98c7-f4f4d628f02e"). InnerVolumeSpecName "kube-api-access-wmpq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:20.064485 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.064464 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d546841-6271-4765-98c7-f4f4d628f02e-tls-certs" pod "6d546841-6271-4765-98c7-f4f4d628f02e" (UID: "6d546841-6271-4765-98c7-f4f4d628f02e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:20:20.142261 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.142231 4793 generic.go:356] "Generic (PLEG): container finished" podID="6d546841-6271-4765-98c7-f4f4d628f02e" containerID="c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace" exitCode=0 Oct 30 05:20:20.142463 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.142279 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" Oct 30 05:20:20.142463 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.142304 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" event={"ID":"6d546841-6271-4765-98c7-f4f4d628f02e","Type":"ContainerDied","Data":"c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace"} Oct 30 05:20:20.142463 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.142344 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz" event={"ID":"6d546841-6271-4765-98c7-f4f4d628f02e","Type":"ContainerDied","Data":"9e61e0d4aadde48af7a42cd61c3417abbffa552384491f6fb894781c0b39e071"} Oct 30 05:20:20.142463 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.142364 4793 scope.go:122] "RemoveContainer" containerID="c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace" Oct 30 05:20:20.154691 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.154673 4793 scope.go:122] "RemoveContainer" containerID="c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace" Oct 30 05:20:20.154958 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:20:20.154937 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace\": container with ID starting with c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace not found: ID does not exist" containerID="c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace" Oct 30 05:20:20.155026 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.154963 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace"} err="failed to get container status \"c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace\": rpc error: code = NotFound desc = could not find container \"c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace\": container with ID starting with c991f26059ad3c93842a1456e60e52b3d9997123f9e606f233084e742145aace not found: ID does not exist" Oct 30 05:20:20.163643 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.163616 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d546841-6271-4765-98c7-f4f4d628f02e-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:20:20.163643 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.163641 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmpq8\" (UniqueName: \"kubernetes.io/projected/6d546841-6271-4765-98c7-f4f4d628f02e-kube-api-access-wmpq8\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:20:20.198446 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.198381 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz"] Oct 30 05:20:20.203472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.203448 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-14305626-kserve-router-scheduler-5b9d45dcbf-rfkzz"] Oct 30 05:20:20.843939 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:20.843910 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d546841-6271-4765-98c7-f4f4d628f02e" path="/var/lib/kubelet/pods/6d546841-6271-4765-98c7-f4f4d628f02e/volumes" Oct 30 05:20:21.038553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:21.038524 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h"] Oct 30 05:20:21.049708 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:21.049676 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff3af461-8s58h"] Oct 30 05:20:21.475530 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:21.475481 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:22.848773 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:22.848740 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3" path="/var/lib/kubelet/pods/d89a3b1a-e196-4cba-b7b3-3d0f248d2eb3/volumes" Oct 30 05:20:23.373339 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:23.373300 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:25.242363 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:25.242326 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:27.085777 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:27.085715 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:29.032207 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:29.032176 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:31.023192 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:31.023152 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:31.258783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:31.258749 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:33.193752 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:33.193717 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:35.198059 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:35.198023 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:37.081384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:37.081330 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:38.932939 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:38.932896 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:40.754729 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:40.754691 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:41.218081 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:20:41.218045 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:20:42.688732 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:42.688696 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:44.437261 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.437223 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k"] Oct 30 05:20:44.437676 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.437615 4793 cpu_manager.go:414] "containerMap: removing container" podUID="6d546841-6271-4765-98c7-f4f4d628f02e" containerName="main" Oct 30 05:20:44.437676 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.437626 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d546841-6271-4765-98c7-f4f4d628f02e" containerName="main" Oct 30 05:20:44.437676 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.437650 4793 cpu_manager.go:414] "containerMap: removing container" podUID="53146743-4864-492b-b5f4-238ec87050b7" containerName="main" Oct 30 05:20:44.437676 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.437656 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53146743-4864-492b-b5f4-238ec87050b7" containerName="main" Oct 30 05:20:44.437821 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.437726 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="53146743-4864-492b-b5f4-238ec87050b7" containerName="main" Oct 30 05:20:44.437821 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.437739 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d546841-6271-4765-98c7-f4f4d628f02e" containerName="main" Oct 30 05:20:44.440710 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.440690 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:44.462601 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.462565 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:20:44.465250 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.465200 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k"] Oct 30 05:20:44.473379 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.473354 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:20:44.473705 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.473528 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:20:44.473705 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.473604 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:20:44.595012 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.594966 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:44.596072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.596050 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsk7c\" (UniqueName: \"kubernetes.io/projected/761a36c0-39fc-4512-b4db-ab8c788b7504-kube-api-access-nsk7c\") pod \"maas-api-revoke-sub-7353741d-jhw7k\" (UID: \"761a36c0-39fc-4512-b4db-ab8c788b7504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:44.596142 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.596102 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7353741d-jhw7k\" (UID: \"761a36c0-39fc-4512-b4db-ab8c788b7504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:44.596184 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.596178 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7353741d-jhw7k\" (UID: \"761a36c0-39fc-4512-b4db-ab8c788b7504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:44.696958 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.696888 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7353741d-jhw7k\" (UID: \"761a36c0-39fc-4512-b4db-ab8c788b7504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:44.696958 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.696950 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsk7c\" (UniqueName: \"kubernetes.io/projected/761a36c0-39fc-4512-b4db-ab8c788b7504-kube-api-access-nsk7c\") pod \"maas-api-revoke-sub-7353741d-jhw7k\" (UID: \"761a36c0-39fc-4512-b4db-ab8c788b7504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:44.697135 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.696997 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7353741d-jhw7k\" (UID: \"761a36c0-39fc-4512-b4db-ab8c788b7504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:44.697587 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.697562 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7353741d-jhw7k\" (UID: \"761a36c0-39fc-4512-b4db-ab8c788b7504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:44.699270 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.699252 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7353741d-jhw7k\" (UID: \"761a36c0-39fc-4512-b4db-ab8c788b7504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:44.720572 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.720548 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsk7c\" (UniqueName: \"kubernetes.io/projected/761a36c0-39fc-4512-b4db-ab8c788b7504-kube-api-access-nsk7c\") pod \"maas-api-revoke-sub-7353741d-jhw7k\" (UID: \"761a36c0-39fc-4512-b4db-ab8c788b7504\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:44.753332 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:44.753311 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:45.061966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:45.061932 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k"] Oct 30 05:20:45.063815 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:20:45.063782 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761a36c0_39fc_4512_b4db_ab8c788b7504.slice/crio-7f21be27b6e1890c7cf1df27755631feef6ec0b17607555b8b84892eac2cbb65 WatchSource:0}: Error finding container 7f21be27b6e1890c7cf1df27755631feef6ec0b17607555b8b84892eac2cbb65: Status 404 returned error can't find the container with id 7f21be27b6e1890c7cf1df27755631feef6ec0b17607555b8b84892eac2cbb65 Oct 30 05:20:45.276879 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:45.276847 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" event={"ID":"761a36c0-39fc-4512-b4db-ab8c788b7504","Type":"ContainerStarted","Data":"7f21be27b6e1890c7cf1df27755631feef6ec0b17607555b8b84892eac2cbb65"} Oct 30 05:20:46.284284 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:46.284246 4793 generic.go:356] "Generic (PLEG): container finished" podID="761a36c0-39fc-4512-b4db-ab8c788b7504" containerID="2515b285c79b4f274dba3c47881145b24c82b936137a3770f04098e3cf39d4af" exitCode=0 Oct 30 05:20:46.284727 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:46.284334 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" event={"ID":"761a36c0-39fc-4512-b4db-ab8c788b7504","Type":"ContainerDied","Data":"2515b285c79b4f274dba3c47881145b24c82b936137a3770f04098e3cf39d4af"} Oct 30 05:20:46.658182 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:46.658099 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:47.437665 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:47.437643 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:47.624981 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:47.624891 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761a36c0-39fc-4512-b4db-ab8c788b7504-kube-api-access-nsk7c\" (UniqueName: \"kubernetes.io/projected/761a36c0-39fc-4512-b4db-ab8c788b7504-kube-api-access-nsk7c\") pod \"761a36c0-39fc-4512-b4db-ab8c788b7504\" (UID: \"761a36c0-39fc-4512-b4db-ab8c788b7504\") " Oct 30 05:20:47.625145 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:47.625009 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-xks-ca\") pod \"761a36c0-39fc-4512-b4db-ab8c788b7504\" (UID: \"761a36c0-39fc-4512-b4db-ab8c788b7504\") " Oct 30 05:20:47.625145 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:47.625062 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-openshift-ca\") pod \"761a36c0-39fc-4512-b4db-ab8c788b7504\" (UID: \"761a36c0-39fc-4512-b4db-ab8c788b7504\") " Oct 30 05:20:47.625450 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:47.625401 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-openshift-ca" pod "761a36c0-39fc-4512-b4db-ab8c788b7504" (UID: "761a36c0-39fc-4512-b4db-ab8c788b7504"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:20:47.627024 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:47.627001 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761a36c0-39fc-4512-b4db-ab8c788b7504-kube-api-access-nsk7c" pod "761a36c0-39fc-4512-b4db-ab8c788b7504" (UID: "761a36c0-39fc-4512-b4db-ab8c788b7504"). InnerVolumeSpecName "kube-api-access-nsk7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:20:47.627134 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:47.627018 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-xks-ca" pod "761a36c0-39fc-4512-b4db-ab8c788b7504" (UID: "761a36c0-39fc-4512-b4db-ab8c788b7504"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:20:47.726764 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:47.726711 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:20:47.726764 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:47.726767 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/761a36c0-39fc-4512-b4db-ab8c788b7504-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:20:47.726988 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:47.726778 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsk7c\" (UniqueName: \"kubernetes.io/projected/761a36c0-39fc-4512-b4db-ab8c788b7504-kube-api-access-nsk7c\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:20:48.296660 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:48.296630 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" Oct 30 05:20:48.296852 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:48.296658 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k" event={"ID":"761a36c0-39fc-4512-b4db-ab8c788b7504","Type":"ContainerDied","Data":"7f21be27b6e1890c7cf1df27755631feef6ec0b17607555b8b84892eac2cbb65"} Oct 30 05:20:48.296852 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:48.296707 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f21be27b6e1890c7cf1df27755631feef6ec0b17607555b8b84892eac2cbb65" Oct 30 05:20:48.556703 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:48.556615 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:50.496724 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:50.496683 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:52.468870 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:52.468836 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:54.382805 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:54.382772 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:55.478198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:55.478163 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:56.037897 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:56.037861 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:56.330692 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:56.330595 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:20:58.175558 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:20:58.175525 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:00.084785 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:00.084679 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:00.389995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:00.389880 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:00.640331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:00.640230 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:02.024591 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:02.024547 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:03.240022 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:03.239981 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:03.566077 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:03.565975 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:04.029950 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:04.029913 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:04.279784 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:04.279752 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:05.633625 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:05.633586 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:10.750644 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:10.750508 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:11.052531 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:11.052450 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:11.217941 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:21:11.217907 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:21:11.646384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:11.646349 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:11.958576 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:11.958547 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:17.036572 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.036541 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf"] Oct 30 05:21:17.037041 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.036938 4793 cpu_manager.go:414] "containerMap: removing container" podUID="761a36c0-39fc-4512-b4db-ab8c788b7504" containerName="revoke-keys" Oct 30 05:21:17.037041 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.036950 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761a36c0-39fc-4512-b4db-ab8c788b7504" containerName="revoke-keys" Oct 30 05:21:17.037041 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.037031 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="761a36c0-39fc-4512-b4db-ab8c788b7504" containerName="revoke-keys" Oct 30 05:21:17.040213 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.040196 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.070199 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.070173 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d1402746-kserve-self-signed-certs\"" Oct 30 05:21:17.072819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.072792 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf"] Oct 30 05:21:17.107677 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.107649 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tmp-dir\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.107799 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.107703 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-model-cache\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.107799 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.107767 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-home\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.107900 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.107838 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-dshm\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.107943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.107915 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tls-certs\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.107999 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.107957 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8464t\" (UniqueName: \"kubernetes.io/projected/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-kube-api-access-8464t\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.209299 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.209267 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tmp-dir\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.209492 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.209315 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-model-cache\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.209492 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.209338 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-home\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.209492 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.209363 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-dshm\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.209492 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.209435 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tls-certs\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.209492 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.209485 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8464t\" (UniqueName: \"kubernetes.io/projected/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-kube-api-access-8464t\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.209813 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.209752 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-model-cache\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.209813 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.209788 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tmp-dir\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.209900 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.209823 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-home\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.211704 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.211682 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-dshm\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.212036 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.212017 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tls-certs\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.265265 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.265227 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8464t\" (UniqueName: \"kubernetes.io/projected/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-kube-api-access-8464t\") pod \"e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.352060 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.351986 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:17.352221 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.352076 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:17.678589 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:17.678565 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf"] Oct 30 05:21:17.680401 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:21:17.680375 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95ac1e2d_e180_4eb1_8270_4eb50fb432f4.slice/crio-738c65375d564b009bec9d00569225ac296cb746c93db85ada8100ba70b3feef WatchSource:0}: Error finding container 738c65375d564b009bec9d00569225ac296cb746c93db85ada8100ba70b3feef: Status 404 returned error can't find the container with id 738c65375d564b009bec9d00569225ac296cb746c93db85ada8100ba70b3feef Oct 30 05:21:18.475250 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:18.475214 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" event={"ID":"95ac1e2d-e180-4eb1-8270-4eb50fb432f4","Type":"ContainerStarted","Data":"3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940"} Oct 30 05:21:18.475250 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:18.475253 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" event={"ID":"95ac1e2d-e180-4eb1-8270-4eb50fb432f4","Type":"ContainerStarted","Data":"738c65375d564b009bec9d00569225ac296cb746c93db85ada8100ba70b3feef"} Oct 30 05:21:18.521323 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:18.521272 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" podStartSLOduration=1.521258674 podStartE2EDuration="1.521258674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:21:18.519352065 +0000 UTC m=+5001.805226840" watchObservedRunningTime="2026-10-30 05:21:18.521258674 +0000 UTC m=+5001.807133449" Oct 30 05:21:19.352793 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:19.352758 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:27.352944 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:27.352911 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:27.365120 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:27.365094 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:27.567695 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:27.567661 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:21:34.152564 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:34.152528 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:39.052755 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:39.052722 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:39.546875 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:39.546834 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:41.218151 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:21:41.218113 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:21:46.032004 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:46.031970 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:21:59.825250 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:59.825209 4793 scope.go:122] "RemoveContainer" containerID="89bdde8540de40ec9af8f3d31e9aad441fe4fdbdb4a1b28efa7df97aec4b759e" Oct 30 05:21:59.851835 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:59.851814 4793 scope.go:122] "RemoveContainer" containerID="43323a2877bf62f0ebf43a22f82d4536c7220e9f6f5ec379f9925a3caca09745" Oct 30 05:21:59.910959 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:21:59.910632 4793 scope.go:122] "RemoveContainer" containerID="a19623f5f3697bb7a8e515adbdaa57c1a2fe1578d331670abe0801aa1286969a" Oct 30 05:22:00.508104 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:00.508060 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:22:08.924612 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:08.924572 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf"] Oct 30 05:22:09.262140 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:09.262100 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:22:09.600690 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:09.600599 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:22:11.218734 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:22:11.218695 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:22:14.502459 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:14.502405 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:22:23.945158 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:23.945093 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" podUID="95ac1e2d-e180-4eb1-8270-4eb50fb432f4" containerName="main" containerID="cri-o://3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940" gracePeriod=45 Oct 30 05:22:24.199675 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.199612 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:22:24.244533 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.244502 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-model-cache\") pod \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " Oct 30 05:22:24.244723 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.244562 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-dshm\") pod \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " Oct 30 05:22:24.244723 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.244667 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tls-certs\") pod \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " Oct 30 05:22:24.244723 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.244698 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tmp-dir\") pod \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " Oct 30 05:22:24.244916 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.244770 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-home\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-home\") pod \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " Oct 30 05:22:24.244916 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.244801 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-kube-api-access-8464t\" (UniqueName: \"kubernetes.io/projected/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-kube-api-access-8464t\") pod \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\" (UID: \"95ac1e2d-e180-4eb1-8270-4eb50fb432f4\") " Oct 30 05:22:24.245623 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.245554 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-home" pod "95ac1e2d-e180-4eb1-8270-4eb50fb432f4" (UID: "95ac1e2d-e180-4eb1-8270-4eb50fb432f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:24.245817 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.245717 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-model-cache" pod "95ac1e2d-e180-4eb1-8270-4eb50fb432f4" (UID: "95ac1e2d-e180-4eb1-8270-4eb50fb432f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:24.246295 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.246251 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tmp-dir" pod "95ac1e2d-e180-4eb1-8270-4eb50fb432f4" (UID: "95ac1e2d-e180-4eb1-8270-4eb50fb432f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:24.247689 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.247664 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-kube-api-access-8464t" pod "95ac1e2d-e180-4eb1-8270-4eb50fb432f4" (UID: "95ac1e2d-e180-4eb1-8270-4eb50fb432f4"). InnerVolumeSpecName "kube-api-access-8464t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:22:24.247804 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.247776 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tls-certs" pod "95ac1e2d-e180-4eb1-8270-4eb50fb432f4" (UID: "95ac1e2d-e180-4eb1-8270-4eb50fb432f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:22:24.247854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.247812 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-dshm" pod "95ac1e2d-e180-4eb1-8270-4eb50fb432f4" (UID: "95ac1e2d-e180-4eb1-8270-4eb50fb432f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:22:24.345741 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.345709 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:22:24.345896 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.345756 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:22:24.345896 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.345765 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:22:24.345896 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.345778 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8464t\" (UniqueName: \"kubernetes.io/projected/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-kube-api-access-8464t\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:22:24.345896 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.345787 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:22:24.345896 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.345795 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/95ac1e2d-e180-4eb1-8270-4eb50fb432f4-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:22:24.949808 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.949774 4793 generic.go:356] "Generic (PLEG): container finished" podID="95ac1e2d-e180-4eb1-8270-4eb50fb432f4" containerID="3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940" exitCode=0 Oct 30 05:22:24.950237 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.949816 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" Oct 30 05:22:24.950237 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.949854 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" event={"ID":"95ac1e2d-e180-4eb1-8270-4eb50fb432f4","Type":"ContainerDied","Data":"3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940"} Oct 30 05:22:24.950237 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.949894 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf" event={"ID":"95ac1e2d-e180-4eb1-8270-4eb50fb432f4","Type":"ContainerDied","Data":"738c65375d564b009bec9d00569225ac296cb746c93db85ada8100ba70b3feef"} Oct 30 05:22:24.950237 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.949911 4793 scope.go:122] "RemoveContainer" containerID="3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940" Oct 30 05:22:24.959521 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.959495 4793 scope.go:122] "RemoveContainer" containerID="3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940" Oct 30 05:22:24.959771 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:22:24.959748 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940\": container with ID starting with 3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940 not found: ID does not exist" containerID="3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940" Oct 30 05:22:24.959839 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:24.959783 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940"} err="failed to get container status \"3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940\": rpc error: code = NotFound desc = could not find container \"3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940\": container with ID starting with 3b137a95cb2838d193b88e3d2b72a0b0df05595b7608e38fc456abfa2ef04940 not found: ID does not exist" Oct 30 05:22:25.004336 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:25.004312 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf"] Oct 30 05:22:25.015955 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:25.015932 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d1402746-kserve-76485f7849-2kcbf"] Oct 30 05:22:26.844525 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:26.844490 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ac1e2d-e180-4eb1-8270-4eb50fb432f4" path="/var/lib/kubelet/pods/95ac1e2d-e180-4eb1-8270-4eb50fb432f4/volumes" Oct 30 05:22:32.035898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:32.035867 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg"] Oct 30 05:22:32.047504 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:32.047478 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-hfgdg"] Oct 30 05:22:32.845735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:32.845660 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec27b38-191a-4560-a8d4-7d8f59d76e2e" path="/var/lib/kubelet/pods/dec27b38-191a-4560-a8d4-7d8f59d76e2e/volumes" Oct 30 05:22:41.218277 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:22:41.218200 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:22:54.902721 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:54.902687 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4"] Oct 30 05:22:54.903155 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:54.903071 4793 cpu_manager.go:414] "containerMap: removing container" podUID="95ac1e2d-e180-4eb1-8270-4eb50fb432f4" containerName="main" Oct 30 05:22:54.903155 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:54.903081 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ac1e2d-e180-4eb1-8270-4eb50fb432f4" containerName="main" Oct 30 05:22:54.903155 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:54.903135 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ac1e2d-e180-4eb1-8270-4eb50fb432f4" containerName="main" Oct 30 05:22:54.906386 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:54.906369 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" Oct 30 05:22:54.928646 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:54.928621 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4"] Oct 30 05:22:54.959372 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:54.959349 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2khs\" (UniqueName: \"kubernetes.io/projected/8b1bd54c-d13a-4a09-a79d-adc3004d7510-kube-api-access-m2khs\") pod \"payload-pre-processing-5c58448fd8-7pcq4\" (UID: \"8b1bd54c-d13a-4a09-a79d-adc3004d7510\") " pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" Oct 30 05:22:54.959530 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:54.959395 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1bd54c-d13a-4a09-a79d-adc3004d7510-plugins-config-volume\") pod \"payload-pre-processing-5c58448fd8-7pcq4\" (UID: \"8b1bd54c-d13a-4a09-a79d-adc3004d7510\") " pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" Oct 30 05:22:55.060635 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:55.060605 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2khs\" (UniqueName: \"kubernetes.io/projected/8b1bd54c-d13a-4a09-a79d-adc3004d7510-kube-api-access-m2khs\") pod \"payload-pre-processing-5c58448fd8-7pcq4\" (UID: \"8b1bd54c-d13a-4a09-a79d-adc3004d7510\") " pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" Oct 30 05:22:55.060776 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:55.060653 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1bd54c-d13a-4a09-a79d-adc3004d7510-plugins-config-volume\") pod \"payload-pre-processing-5c58448fd8-7pcq4\" (UID: \"8b1bd54c-d13a-4a09-a79d-adc3004d7510\") " pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" Oct 30 05:22:55.061466 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:55.061444 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1bd54c-d13a-4a09-a79d-adc3004d7510-plugins-config-volume\") pod \"payload-pre-processing-5c58448fd8-7pcq4\" (UID: \"8b1bd54c-d13a-4a09-a79d-adc3004d7510\") " pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" Oct 30 05:22:55.087758 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:55.087737 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2khs\" (UniqueName: \"kubernetes.io/projected/8b1bd54c-d13a-4a09-a79d-adc3004d7510-kube-api-access-m2khs\") pod \"payload-pre-processing-5c58448fd8-7pcq4\" (UID: \"8b1bd54c-d13a-4a09-a79d-adc3004d7510\") " pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" Oct 30 05:22:55.219648 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:55.219616 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" Oct 30 05:22:55.523821 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:55.523786 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4"] Oct 30 05:22:55.524907 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:22:55.524884 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1bd54c_d13a_4a09_a79d_adc3004d7510.slice/crio-5afad1a8d4ccee343902fee626b915d438029b07b75e06d8dec087b7a8d4758d WatchSource:0}: Error finding container 5afad1a8d4ccee343902fee626b915d438029b07b75e06d8dec087b7a8d4758d: Status 404 returned error can't find the container with id 5afad1a8d4ccee343902fee626b915d438029b07b75e06d8dec087b7a8d4758d Oct 30 05:22:56.033706 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:56.033675 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx"] Oct 30 05:22:56.045209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:56.045176 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-xrdfx"] Oct 30 05:22:56.150031 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:56.149995 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" event={"ID":"8b1bd54c-d13a-4a09-a79d-adc3004d7510","Type":"ContainerStarted","Data":"5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1"} Oct 30 05:22:56.150192 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:56.150059 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" event={"ID":"8b1bd54c-d13a-4a09-a79d-adc3004d7510","Type":"ContainerStarted","Data":"5afad1a8d4ccee343902fee626b915d438029b07b75e06d8dec087b7a8d4758d"} Oct 30 05:22:56.150192 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:56.150087 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" Oct 30 05:22:56.197898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:56.197852 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" podStartSLOduration=2.197839983 podStartE2EDuration="2.197839983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:22:56.194024448 +0000 UTC m=+5099.479899243" watchObservedRunningTime="2026-10-30 05:22:56.197839983 +0000 UTC m=+5099.483714752" Oct 30 05:22:56.851914 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:22:56.851882 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a49595-22f0-42d5-9e92-ba8920518f83" path="/var/lib/kubelet/pods/73a49595-22f0-42d5-9e92-ba8920518f83/volumes" Oct 30 05:23:00.018126 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:00.018094 4793 scope.go:122] "RemoveContainer" containerID="a58583b3c5e00d6b0215364aec34a77b0624950c31fe5084bf497de7ffeb4e2b" Oct 30 05:23:00.679552 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:00.679520 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:23:07.158235 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:07.158200 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" Oct 30 05:23:07.243564 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:07.243533 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv"] Oct 30 05:23:09.262476 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:09.262441 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-77bf598fff-7f6b5"] Oct 30 05:23:09.267533 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:09.267511 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" Oct 30 05:23:09.364472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:09.364437 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-77bf598fff-7f6b5"] Oct 30 05:23:09.381985 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:09.381959 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35655153-13ee-4415-b70a-2e89ef47d117-plugins-config-volume\") pod \"payload-processing-77bf598fff-7f6b5\" (UID: \"35655153-13ee-4415-b70a-2e89ef47d117\") " pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" Oct 30 05:23:09.382112 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:09.382081 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdrbz\" (UniqueName: \"kubernetes.io/projected/35655153-13ee-4415-b70a-2e89ef47d117-kube-api-access-bdrbz\") pod \"payload-processing-77bf598fff-7f6b5\" (UID: \"35655153-13ee-4415-b70a-2e89ef47d117\") " pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" Oct 30 05:23:09.483167 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:09.483142 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35655153-13ee-4415-b70a-2e89ef47d117-plugins-config-volume\") pod \"payload-processing-77bf598fff-7f6b5\" (UID: \"35655153-13ee-4415-b70a-2e89ef47d117\") " pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" Oct 30 05:23:09.483295 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:09.483214 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdrbz\" (UniqueName: \"kubernetes.io/projected/35655153-13ee-4415-b70a-2e89ef47d117-kube-api-access-bdrbz\") pod \"payload-processing-77bf598fff-7f6b5\" (UID: \"35655153-13ee-4415-b70a-2e89ef47d117\") " pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" Oct 30 05:23:09.483863 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:09.483844 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35655153-13ee-4415-b70a-2e89ef47d117-plugins-config-volume\") pod \"payload-processing-77bf598fff-7f6b5\" (UID: \"35655153-13ee-4415-b70a-2e89ef47d117\") " pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" Oct 30 05:23:09.508039 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:09.508016 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdrbz\" (UniqueName: \"kubernetes.io/projected/35655153-13ee-4415-b70a-2e89ef47d117-kube-api-access-bdrbz\") pod \"payload-processing-77bf598fff-7f6b5\" (UID: \"35655153-13ee-4415-b70a-2e89ef47d117\") " pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" Oct 30 05:23:09.580662 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:09.580603 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" Oct 30 05:23:09.883409 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:09.883380 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-77bf598fff-7f6b5"] Oct 30 05:23:09.884045 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:23:09.884019 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35655153_13ee_4415_b70a_2e89ef47d117.slice/crio-59b4675e9e8d4c7e78249484b938abc009deba7a31168729002aeeb4331f2576 WatchSource:0}: Error finding container 59b4675e9e8d4c7e78249484b938abc009deba7a31168729002aeeb4331f2576: Status 404 returned error can't find the container with id 59b4675e9e8d4c7e78249484b938abc009deba7a31168729002aeeb4331f2576 Oct 30 05:23:10.229258 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:10.229220 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" event={"ID":"35655153-13ee-4415-b70a-2e89ef47d117","Type":"ContainerStarted","Data":"b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58"} Oct 30 05:23:10.229467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:10.229287 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" Oct 30 05:23:10.229467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:10.229310 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" event={"ID":"35655153-13ee-4415-b70a-2e89ef47d117","Type":"ContainerStarted","Data":"59b4675e9e8d4c7e78249484b938abc009deba7a31168729002aeeb4331f2576"} Oct 30 05:23:10.281548 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:10.281496 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" podStartSLOduration=1.281483365 podStartE2EDuration="1.281483365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:23:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:23:10.277323637 +0000 UTC m=+5113.563198411" watchObservedRunningTime="2026-10-30 05:23:10.281483365 +0000 UTC m=+5113.567358140" Oct 30 05:23:11.219327 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:23:11.219292 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:23:12.037276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:12.037240 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z"] Oct 30 05:23:12.048111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:12.048085 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-hng5z"] Oct 30 05:23:12.244665 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:12.244632 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" podUID="d09ef3d4-79e1-4a47-8b15-0b863ef31ff3" containerName="payload-pre-processing" containerID="cri-o://a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c" gracePeriod=25 Oct 30 05:23:12.498216 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:12.498192 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" Oct 30 05:23:12.613055 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:12.612981 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-kube-api-access-6vn6g\" (UniqueName: \"kubernetes.io/projected/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-kube-api-access-6vn6g\") pod \"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3\" (UID: \"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3\") " Oct 30 05:23:12.613201 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:12.613066 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-plugins-config-volume\") pod \"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3\" (UID: \"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3\") " Oct 30 05:23:12.613413 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:12.613394 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-plugins-config-volume" pod "d09ef3d4-79e1-4a47-8b15-0b863ef31ff3" (UID: "d09ef3d4-79e1-4a47-8b15-0b863ef31ff3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:12.615064 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:12.615044 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-kube-api-access-6vn6g" pod "d09ef3d4-79e1-4a47-8b15-0b863ef31ff3" (UID: "d09ef3d4-79e1-4a47-8b15-0b863ef31ff3"). InnerVolumeSpecName "kube-api-access-6vn6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:12.714331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:12.714297 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vn6g\" (UniqueName: \"kubernetes.io/projected/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-kube-api-access-6vn6g\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:23:12.714331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:12.714320 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:23:12.845444 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:12.845404 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b34c9d7e-8403-40d6-a187-176c465b85a3" path="/var/lib/kubelet/pods/b34c9d7e-8403-40d6-a187-176c465b85a3/volumes" Oct 30 05:23:12.883033 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:23:12.882971 4793 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09ef3d4_79e1_4a47_8b15_0b863ef31ff3.slice/crio-1d2b0f4ebe11a3d931e909e4654c72e7c9e5df81dcd9f299a586672030b062b0\": RecentStats: unable to find data in memory cache]" Oct 30 05:23:13.245151 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:13.245116 4793 generic.go:356] "Generic (PLEG): container finished" podID="d09ef3d4-79e1-4a47-8b15-0b863ef31ff3" containerID="a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c" exitCode=0 Oct 30 05:23:13.245637 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:13.245175 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" Oct 30 05:23:13.245637 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:13.245205 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" event={"ID":"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3","Type":"ContainerDied","Data":"a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c"} Oct 30 05:23:13.245637 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:13.245303 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv" event={"ID":"d09ef3d4-79e1-4a47-8b15-0b863ef31ff3","Type":"ContainerDied","Data":"1d2b0f4ebe11a3d931e909e4654c72e7c9e5df81dcd9f299a586672030b062b0"} Oct 30 05:23:13.245637 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:13.245338 4793 scope.go:122] "RemoveContainer" containerID="a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c" Oct 30 05:23:13.255004 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:13.254984 4793 scope.go:122] "RemoveContainer" containerID="a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c" Oct 30 05:23:13.255259 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:23:13.255238 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c\": container with ID starting with a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c not found: ID does not exist" containerID="a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c" Oct 30 05:23:13.255310 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:13.255268 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c"} err="failed to get container status \"a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c\": rpc error: code = NotFound desc = could not find container \"a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c\": container with ID starting with a25bda0cc09d06cb573e95128af1964ecf0084642808ac941b4c95108ab44a3c not found: ID does not exist" Oct 30 05:23:13.296674 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:13.296648 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv"] Oct 30 05:23:13.308229 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:13.308207 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-v7lnv"] Oct 30 05:23:14.843736 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:14.843705 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09ef3d4-79e1-4a47-8b15-0b863ef31ff3" path="/var/lib/kubelet/pods/d09ef3d4-79e1-4a47-8b15-0b863ef31ff3/volumes" Oct 30 05:23:21.237776 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:21.237742 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" Oct 30 05:23:21.315076 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:21.315048 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-l4r5r"] Oct 30 05:23:23.033965 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.033932 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d"] Oct 30 05:23:23.045772 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.045743 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-78n5d"] Oct 30 05:23:23.318356 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.318278 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5688bd49d7-w75hb"] Oct 30 05:23:23.318686 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.318673 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d09ef3d4-79e1-4a47-8b15-0b863ef31ff3" containerName="payload-pre-processing" Oct 30 05:23:23.318738 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.318688 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09ef3d4-79e1-4a47-8b15-0b863ef31ff3" containerName="payload-pre-processing" Oct 30 05:23:23.318775 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.318753 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09ef3d4-79e1-4a47-8b15-0b863ef31ff3" containerName="payload-pre-processing" Oct 30 05:23:23.323325 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.323305 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" Oct 30 05:23:23.365254 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.365233 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5688bd49d7-w75hb"] Oct 30 05:23:23.414765 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.414742 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4815527-3e97-4f0b-bd06-be6277725976-plugins-config-volume\") pod \"payload-processing-5688bd49d7-w75hb\" (UID: \"e4815527-3e97-4f0b-bd06-be6277725976\") " pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" Oct 30 05:23:23.414890 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.414799 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qsw\" (UniqueName: \"kubernetes.io/projected/e4815527-3e97-4f0b-bd06-be6277725976-kube-api-access-l8qsw\") pod \"payload-processing-5688bd49d7-w75hb\" (UID: \"e4815527-3e97-4f0b-bd06-be6277725976\") " pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" Oct 30 05:23:23.515994 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.515961 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qsw\" (UniqueName: \"kubernetes.io/projected/e4815527-3e97-4f0b-bd06-be6277725976-kube-api-access-l8qsw\") pod \"payload-processing-5688bd49d7-w75hb\" (UID: \"e4815527-3e97-4f0b-bd06-be6277725976\") " pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" Oct 30 05:23:23.516137 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.516080 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4815527-3e97-4f0b-bd06-be6277725976-plugins-config-volume\") pod \"payload-processing-5688bd49d7-w75hb\" (UID: \"e4815527-3e97-4f0b-bd06-be6277725976\") " pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" Oct 30 05:23:23.516828 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.516809 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4815527-3e97-4f0b-bd06-be6277725976-plugins-config-volume\") pod \"payload-processing-5688bd49d7-w75hb\" (UID: \"e4815527-3e97-4f0b-bd06-be6277725976\") " pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" Oct 30 05:23:23.541324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.541303 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qsw\" (UniqueName: \"kubernetes.io/projected/e4815527-3e97-4f0b-bd06-be6277725976-kube-api-access-l8qsw\") pod \"payload-processing-5688bd49d7-w75hb\" (UID: \"e4815527-3e97-4f0b-bd06-be6277725976\") " pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" Oct 30 05:23:23.636008 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.635934 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" Oct 30 05:23:23.939352 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:23:23.939323 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4815527_3e97_4f0b_bd06_be6277725976.slice/crio-76b020feae49b2eda752ef7a30f26ea9677a4a539b933fd18572c8720c051852 WatchSource:0}: Error finding container 76b020feae49b2eda752ef7a30f26ea9677a4a539b933fd18572c8720c051852: Status 404 returned error can't find the container with id 76b020feae49b2eda752ef7a30f26ea9677a4a539b933fd18572c8720c051852 Oct 30 05:23:23.939884 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:23.939837 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5688bd49d7-w75hb"] Oct 30 05:23:24.307450 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:24.307340 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" event={"ID":"e4815527-3e97-4f0b-bd06-be6277725976","Type":"ContainerStarted","Data":"18cd1eeee11b6ee4aa389354995123149406eac72c68b20df62e08fa8b668c1f"} Oct 30 05:23:24.307450 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:24.307443 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" event={"ID":"e4815527-3e97-4f0b-bd06-be6277725976","Type":"ContainerStarted","Data":"76b020feae49b2eda752ef7a30f26ea9677a4a539b933fd18572c8720c051852"} Oct 30 05:23:24.307863 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:24.307473 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" Oct 30 05:23:24.351107 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:24.351024 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" podStartSLOduration=1.351010746 podStartE2EDuration="1.351010746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:23:24.349520143 +0000 UTC m=+5127.635394932" watchObservedRunningTime="2026-10-30 05:23:24.351010746 +0000 UTC m=+5127.636885521" Oct 30 05:23:24.844800 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:24.844760 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d1e89c-683f-4dcc-b71c-240b846494cc" path="/var/lib/kubelet/pods/33d1e89c-683f-4dcc-b71c-240b846494cc/volumes" Oct 30 05:23:26.315497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:26.315456 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" podUID="db99ac60-acfe-44f7-9729-9ef0ef451830" containerName="payload-processing" containerID="cri-o://0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0" gracePeriod=25 Oct 30 05:23:26.562715 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:26.562691 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" Oct 30 05:23:26.643338 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:26.643264 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db99ac60-acfe-44f7-9729-9ef0ef451830-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db99ac60-acfe-44f7-9729-9ef0ef451830-plugins-config-volume\") pod \"db99ac60-acfe-44f7-9729-9ef0ef451830\" (UID: \"db99ac60-acfe-44f7-9729-9ef0ef451830\") " Oct 30 05:23:26.643493 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:26.643465 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db99ac60-acfe-44f7-9729-9ef0ef451830-kube-api-access-79ddd\" (UniqueName: \"kubernetes.io/projected/db99ac60-acfe-44f7-9729-9ef0ef451830-kube-api-access-79ddd\") pod \"db99ac60-acfe-44f7-9729-9ef0ef451830\" (UID: \"db99ac60-acfe-44f7-9729-9ef0ef451830\") " Oct 30 05:23:26.643643 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:26.643623 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db99ac60-acfe-44f7-9729-9ef0ef451830-plugins-config-volume" pod "db99ac60-acfe-44f7-9729-9ef0ef451830" (UID: "db99ac60-acfe-44f7-9729-9ef0ef451830"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:26.643876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:26.643856 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db99ac60-acfe-44f7-9729-9ef0ef451830-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:23:26.645677 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:26.645653 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db99ac60-acfe-44f7-9729-9ef0ef451830-kube-api-access-79ddd" pod "db99ac60-acfe-44f7-9729-9ef0ef451830" (UID: "db99ac60-acfe-44f7-9729-9ef0ef451830"). InnerVolumeSpecName "kube-api-access-79ddd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:26.744307 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:26.744281 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79ddd\" (UniqueName: \"kubernetes.io/projected/db99ac60-acfe-44f7-9729-9ef0ef451830-kube-api-access-79ddd\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:23:27.324445 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:27.324400 4793 generic.go:356] "Generic (PLEG): container finished" podID="db99ac60-acfe-44f7-9729-9ef0ef451830" containerID="0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0" exitCode=0 Oct 30 05:23:27.324858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:27.324452 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" event={"ID":"db99ac60-acfe-44f7-9729-9ef0ef451830","Type":"ContainerDied","Data":"0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0"} Oct 30 05:23:27.324858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:27.324489 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" event={"ID":"db99ac60-acfe-44f7-9729-9ef0ef451830","Type":"ContainerDied","Data":"f59a6742ef50c0c49302dfeb0a74c1deba267363660a4ca183ca57b3e5a1d122"} Oct 30 05:23:27.324858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:27.324508 4793 scope.go:122] "RemoveContainer" containerID="0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0" Oct 30 05:23:27.324858 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:27.324453 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-l4r5r" Oct 30 05:23:27.333760 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:27.333738 4793 scope.go:122] "RemoveContainer" containerID="0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0" Oct 30 05:23:27.334031 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:23:27.334012 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0\": container with ID starting with 0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0 not found: ID does not exist" containerID="0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0" Oct 30 05:23:27.334080 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:27.334040 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0"} err="failed to get container status \"0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0\": rpc error: code = NotFound desc = could not find container \"0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0\": container with ID starting with 0a79590d4aed0b9cddf218056403d45e884e7c64fa7749bbbb79fe01ca9827b0 not found: ID does not exist" Oct 30 05:23:27.381075 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:27.381044 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-l4r5r"] Oct 30 05:23:27.392397 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:27.392375 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-l4r5r"] Oct 30 05:23:28.844634 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:28.844601 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db99ac60-acfe-44f7-9729-9ef0ef451830" path="/var/lib/kubelet/pods/db99ac60-acfe-44f7-9729-9ef0ef451830/volumes" Oct 30 05:23:35.316534 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:35.316503 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5688bd49d7-w75hb" Oct 30 05:23:35.390309 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:35.390278 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-77bf598fff-7f6b5"] Oct 30 05:23:35.999348 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:35.999310 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4"] Oct 30 05:23:38.259610 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:38.259568 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:23:38.653244 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:38.653166 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:23:39.178022 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.177985 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:23:39.321329 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.321287 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-bpnfm"] Oct 30 05:23:39.321899 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.321878 4793 cpu_manager.go:414] "containerMap: removing container" podUID="db99ac60-acfe-44f7-9729-9ef0ef451830" containerName="payload-processing" Oct 30 05:23:39.321992 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.321901 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db99ac60-acfe-44f7-9729-9ef0ef451830" containerName="payload-processing" Oct 30 05:23:39.322049 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.322001 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="db99ac60-acfe-44f7-9729-9ef0ef451830" containerName="payload-processing" Oct 30 05:23:39.326035 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.326012 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.353776 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.353751 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Oct 30 05:23:39.353776 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.353768 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-nvfmj\"" Oct 30 05:23:39.356583 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.356560 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-bpnfm"] Oct 30 05:23:39.370123 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.370094 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-plugins-config-volume\") pod \"payload-processing-external-model-758758b45d-bpnfm\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.370258 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.370163 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9x5\" (UniqueName: \"kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-kube-api-access-vl9x5\") pod \"payload-processing-external-model-758758b45d-bpnfm\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.370540 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.370519 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-routing-overlay-volume\") pod \"payload-processing-external-model-758758b45d-bpnfm\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.370620 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.370556 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-provider-credentials\") pod \"payload-processing-external-model-758758b45d-bpnfm\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.429920 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.429831 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 05:23:39.430182 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.429830 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 05:23:39.430182 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.429919 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 05:23:39.430392 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.429974 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 05:23:39.471183 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.471140 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9x5\" (UniqueName: \"kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-kube-api-access-vl9x5\") pod \"payload-processing-external-model-758758b45d-bpnfm\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.471347 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.471198 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-routing-overlay-volume\") pod \"payload-processing-external-model-758758b45d-bpnfm\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.471347 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.471245 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-provider-credentials\") pod \"payload-processing-external-model-758758b45d-bpnfm\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.471505 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.471369 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-plugins-config-volume\") pod \"payload-processing-external-model-758758b45d-bpnfm\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.472214 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.472183 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-routing-overlay-volume\") pod \"payload-processing-external-model-758758b45d-bpnfm\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.472395 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.472273 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-plugins-config-volume\") pod \"payload-processing-external-model-758758b45d-bpnfm\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.474450 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.474398 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-provider-credentials\") pod \"payload-processing-external-model-758758b45d-bpnfm\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.500015 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.499991 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9x5\" (UniqueName: \"kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-kube-api-access-vl9x5\") pod \"payload-processing-external-model-758758b45d-bpnfm\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.638718 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.638690 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:39.948047 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:39.948013 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-bpnfm"] Oct 30 05:23:39.949203 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:23:39.949176 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951ef5dd_a558_4b58_b22c_b7ffcf625d1c.slice/crio-eb42e8dcc24bc5f235b5aee7326b99b762a0d9c381823cb69df9d9d4ade9159e WatchSource:0}: Error finding container eb42e8dcc24bc5f235b5aee7326b99b762a0d9c381823cb69df9d9d4ade9159e: Status 404 returned error can't find the container with id eb42e8dcc24bc5f235b5aee7326b99b762a0d9c381823cb69df9d9d4ade9159e Oct 30 05:23:40.391391 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.391282 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" podUID="35655153-13ee-4415-b70a-2e89ef47d117" containerName="payload-processing" containerID="cri-o://b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58" gracePeriod=25 Oct 30 05:23:40.397484 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.397441 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" event={"ID":"951ef5dd-a558-4b58-b22c-b7ffcf625d1c","Type":"ContainerStarted","Data":"9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f"} Oct 30 05:23:40.397624 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.397500 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" event={"ID":"951ef5dd-a558-4b58-b22c-b7ffcf625d1c","Type":"ContainerStarted","Data":"eb42e8dcc24bc5f235b5aee7326b99b762a0d9c381823cb69df9d9d4ade9159e"} Oct 30 05:23:40.397624 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.397549 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:23:40.442494 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.442447 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" podStartSLOduration=1.442432805 podStartE2EDuration="1.442432805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:23:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:23:40.440577752 +0000 UTC m=+5143.726452531" watchObservedRunningTime="2026-10-30 05:23:40.442432805 +0000 UTC m=+5143.728307571" Oct 30 05:23:40.651182 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.651124 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" Oct 30 05:23:40.685093 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.685066 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35655153-13ee-4415-b70a-2e89ef47d117-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35655153-13ee-4415-b70a-2e89ef47d117-plugins-config-volume\") pod \"35655153-13ee-4415-b70a-2e89ef47d117\" (UID: \"35655153-13ee-4415-b70a-2e89ef47d117\") " Oct 30 05:23:40.685288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.685272 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35655153-13ee-4415-b70a-2e89ef47d117-kube-api-access-bdrbz\" (UniqueName: \"kubernetes.io/projected/35655153-13ee-4415-b70a-2e89ef47d117-kube-api-access-bdrbz\") pod \"35655153-13ee-4415-b70a-2e89ef47d117\" (UID: \"35655153-13ee-4415-b70a-2e89ef47d117\") " Oct 30 05:23:40.685537 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.685511 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35655153-13ee-4415-b70a-2e89ef47d117-plugins-config-volume" pod "35655153-13ee-4415-b70a-2e89ef47d117" (UID: "35655153-13ee-4415-b70a-2e89ef47d117"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:40.687343 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.687314 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35655153-13ee-4415-b70a-2e89ef47d117-kube-api-access-bdrbz" pod "35655153-13ee-4415-b70a-2e89ef47d117" (UID: "35655153-13ee-4415-b70a-2e89ef47d117"). InnerVolumeSpecName "kube-api-access-bdrbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:40.786600 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.786571 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdrbz\" (UniqueName: \"kubernetes.io/projected/35655153-13ee-4415-b70a-2e89ef47d117-kube-api-access-bdrbz\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:23:40.786600 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.786599 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35655153-13ee-4415-b70a-2e89ef47d117-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:23:41.000038 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:40.999988 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" podUID="8b1bd54c-d13a-4a09-a79d-adc3004d7510" containerName="payload-pre-processing" containerID="cri-o://5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1" gracePeriod=25 Oct 30 05:23:41.224999 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:23:41.224964 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:23:41.235807 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.235786 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" Oct 30 05:23:41.291996 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.291928 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1bd54c-d13a-4a09-a79d-adc3004d7510-kube-api-access-m2khs\" (UniqueName: \"kubernetes.io/projected/8b1bd54c-d13a-4a09-a79d-adc3004d7510-kube-api-access-m2khs\") pod \"8b1bd54c-d13a-4a09-a79d-adc3004d7510\" (UID: \"8b1bd54c-d13a-4a09-a79d-adc3004d7510\") " Oct 30 05:23:41.292127 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.292042 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1bd54c-d13a-4a09-a79d-adc3004d7510-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1bd54c-d13a-4a09-a79d-adc3004d7510-plugins-config-volume\") pod \"8b1bd54c-d13a-4a09-a79d-adc3004d7510\" (UID: \"8b1bd54c-d13a-4a09-a79d-adc3004d7510\") " Oct 30 05:23:41.292482 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.292455 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1bd54c-d13a-4a09-a79d-adc3004d7510-plugins-config-volume" pod "8b1bd54c-d13a-4a09-a79d-adc3004d7510" (UID: "8b1bd54c-d13a-4a09-a79d-adc3004d7510"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:23:41.293947 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.293923 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1bd54c-d13a-4a09-a79d-adc3004d7510-kube-api-access-m2khs" pod "8b1bd54c-d13a-4a09-a79d-adc3004d7510" (UID: "8b1bd54c-d13a-4a09-a79d-adc3004d7510"). InnerVolumeSpecName "kube-api-access-m2khs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:23:41.393107 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.393068 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2khs\" (UniqueName: \"kubernetes.io/projected/8b1bd54c-d13a-4a09-a79d-adc3004d7510-kube-api-access-m2khs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:23:41.393107 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.393099 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b1bd54c-d13a-4a09-a79d-adc3004d7510-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:23:41.404805 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.404776 4793 generic.go:356] "Generic (PLEG): container finished" podID="35655153-13ee-4415-b70a-2e89ef47d117" containerID="b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58" exitCode=0 Oct 30 05:23:41.404946 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.404827 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" Oct 30 05:23:41.404946 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.404865 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" event={"ID":"35655153-13ee-4415-b70a-2e89ef47d117","Type":"ContainerDied","Data":"b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58"} Oct 30 05:23:41.404946 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.404917 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-77bf598fff-7f6b5" event={"ID":"35655153-13ee-4415-b70a-2e89ef47d117","Type":"ContainerDied","Data":"59b4675e9e8d4c7e78249484b938abc009deba7a31168729002aeeb4331f2576"} Oct 30 05:23:41.405077 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.404946 4793 scope.go:122] "RemoveContainer" containerID="b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58" Oct 30 05:23:41.406518 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.406494 4793 generic.go:356] "Generic (PLEG): container finished" podID="8b1bd54c-d13a-4a09-a79d-adc3004d7510" containerID="5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1" exitCode=0 Oct 30 05:23:41.406656 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.406538 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" Oct 30 05:23:41.406656 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.406563 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" event={"ID":"8b1bd54c-d13a-4a09-a79d-adc3004d7510","Type":"ContainerDied","Data":"5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1"} Oct 30 05:23:41.406656 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.406629 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4" event={"ID":"8b1bd54c-d13a-4a09-a79d-adc3004d7510","Type":"ContainerDied","Data":"5afad1a8d4ccee343902fee626b915d438029b07b75e06d8dec087b7a8d4758d"} Oct 30 05:23:41.415739 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.415721 4793 scope.go:122] "RemoveContainer" containerID="b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58" Oct 30 05:23:41.416183 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:23:41.415975 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58\": container with ID starting with b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58 not found: ID does not exist" containerID="b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58" Oct 30 05:23:41.416183 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.416000 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58"} err="failed to get container status \"b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58\": rpc error: code = NotFound desc = could not find container \"b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58\": container with ID starting with b0a39eb560ad830e901b8e39c9a154f20f09c54b4883ecdd2130d814131f1c58 not found: ID does not exist" Oct 30 05:23:41.416183 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.416023 4793 scope.go:122] "RemoveContainer" containerID="5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1" Oct 30 05:23:41.437469 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.437410 4793 scope.go:122] "RemoveContainer" containerID="5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1" Oct 30 05:23:41.437815 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:23:41.437785 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1\": container with ID starting with 5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1 not found: ID does not exist" containerID="5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1" Oct 30 05:23:41.437905 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.437828 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1"} err="failed to get container status \"5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1\": rpc error: code = NotFound desc = could not find container \"5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1\": container with ID starting with 5a770ffe943d0da26a063d84535123dd3aa9743341191ce80c4006d410832ac1 not found: ID does not exist" Oct 30 05:23:41.471464 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.471410 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-77bf598fff-7f6b5"] Oct 30 05:23:41.487546 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.487501 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-77bf598fff-7f6b5"] Oct 30 05:23:41.517007 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.516976 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4"] Oct 30 05:23:41.522244 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:41.522213 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-5c58448fd8-7pcq4"] Oct 30 05:23:42.850763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:42.850731 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35655153-13ee-4415-b70a-2e89ef47d117" path="/var/lib/kubelet/pods/35655153-13ee-4415-b70a-2e89ef47d117/volumes" Oct 30 05:23:42.851252 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:42.851137 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1bd54c-d13a-4a09-a79d-adc3004d7510" path="/var/lib/kubelet/pods/8b1bd54c-d13a-4a09-a79d-adc3004d7510/volumes" Oct 30 05:23:51.409664 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:23:51.409632 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:24:00.043046 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:24:00.043012 4793 scope.go:122] "RemoveContainer" containerID="873a2e3aca5925588af598308dcbbae9988927e5a1feb4c4cf2bfc882e5e7788" Oct 30 05:24:00.051498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:24:00.051475 4793 scope.go:122] "RemoveContainer" containerID="1bc57d42fec6e0b80869f49feae702a2ece7f0622a52007b57ce4bffadb8d74a" Oct 30 05:24:00.749955 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:24:00.749920 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:24:11.217964 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:24:11.217874 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:24:41.217570 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:24:41.217531 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:25:00.092943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:00.092907 4793 scope.go:122] "RemoveContainer" containerID="5304ba820fc55768b75119e093d8e086cd0d193dae0bedfbc71644bdb599451f" Oct 30 05:25:00.101646 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:00.101620 4793 scope.go:122] "RemoveContainer" containerID="01c8812f5840befc9ba62b083ff930b6c3843ec3f4981614d734de9dc0abf2e8" Oct 30 05:25:00.889024 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:00.888984 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:25:05.838671 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.838634 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b"] Oct 30 05:25:05.840221 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.840194 4793 cpu_manager.go:414] "containerMap: removing container" podUID="8b1bd54c-d13a-4a09-a79d-adc3004d7510" containerName="payload-pre-processing" Oct 30 05:25:05.840221 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.840218 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1bd54c-d13a-4a09-a79d-adc3004d7510" containerName="payload-pre-processing" Oct 30 05:25:05.840459 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.840231 4793 cpu_manager.go:414] "containerMap: removing container" podUID="35655153-13ee-4415-b70a-2e89ef47d117" containerName="payload-processing" Oct 30 05:25:05.840459 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.840236 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35655153-13ee-4415-b70a-2e89ef47d117" containerName="payload-processing" Oct 30 05:25:05.840459 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.840316 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1bd54c-d13a-4a09-a79d-adc3004d7510" containerName="payload-pre-processing" Oct 30 05:25:05.840459 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.840325 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="35655153-13ee-4415-b70a-2e89ef47d117" containerName="payload-processing" Oct 30 05:25:05.845158 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.845138 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:05.859262 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.859238 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b"] Oct 30 05:25:05.869434 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.869398 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:25:05.869553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.869406 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:25:05.869553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.869453 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:25:05.869553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.869501 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:25:05.967400 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.967370 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j7jk\" (UniqueName: \"kubernetes.io/projected/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-kube-api-access-6j7jk\") pod \"maas-api-revoke-sub-e2547622-xjm2b\" (UID: \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:05.967601 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.967579 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-xjm2b\" (UID: \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:05.967668 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:05.967619 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-xjm2b\" (UID: \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:06.068520 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:06.068492 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j7jk\" (UniqueName: \"kubernetes.io/projected/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-kube-api-access-6j7jk\") pod \"maas-api-revoke-sub-e2547622-xjm2b\" (UID: \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:06.068663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:06.068577 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-xjm2b\" (UID: \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:06.068663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:06.068607 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-xjm2b\" (UID: \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:06.069352 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:06.069333 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-xjm2b\" (UID: \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:06.071010 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:06.070986 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-xjm2b\" (UID: \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:06.091887 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:06.091832 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j7jk\" (UniqueName: \"kubernetes.io/projected/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-kube-api-access-6j7jk\") pod \"maas-api-revoke-sub-e2547622-xjm2b\" (UID: \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:06.157234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:06.157214 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:06.453437 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:06.453395 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b"] Oct 30 05:25:06.456081 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:25:06.456052 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fca5e27_9fda_45d7_83f7_fd0d25147bd1.slice/crio-432c9170bc41163f4a4023a69d7fd4574cb0337b6292e0192062f2ccb95f024f WatchSource:0}: Error finding container 432c9170bc41163f4a4023a69d7fd4574cb0337b6292e0192062f2ccb95f024f: Status 404 returned error can't find the container with id 432c9170bc41163f4a4023a69d7fd4574cb0337b6292e0192062f2ccb95f024f Oct 30 05:25:06.872502 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:06.872409 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" event={"ID":"5fca5e27-9fda-45d7-83f7-fd0d25147bd1","Type":"ContainerStarted","Data":"82960618a8d8e21230c730a1e4314b78209bac60bb922a63cd1b54b54cd09d0a"} Oct 30 05:25:06.872502 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:06.872471 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" event={"ID":"5fca5e27-9fda-45d7-83f7-fd0d25147bd1","Type":"ContainerStarted","Data":"432c9170bc41163f4a4023a69d7fd4574cb0337b6292e0192062f2ccb95f024f"} Oct 30 05:25:06.912513 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:06.912462 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" podStartSLOduration=1.912445803 podStartE2EDuration="1.912445803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:06.911237135 +0000 UTC m=+5230.197111910" watchObservedRunningTime="2026-10-30 05:25:06.912445803 +0000 UTC m=+5230.198320572" Oct 30 05:25:07.878408 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:07.878373 4793 generic.go:356] "Generic (PLEG): container finished" podID="5fca5e27-9fda-45d7-83f7-fd0d25147bd1" containerID="82960618a8d8e21230c730a1e4314b78209bac60bb922a63cd1b54b54cd09d0a" exitCode=0 Oct 30 05:25:07.878833 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:07.878466 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" event={"ID":"5fca5e27-9fda-45d7-83f7-fd0d25147bd1","Type":"ContainerDied","Data":"82960618a8d8e21230c730a1e4314b78209bac60bb922a63cd1b54b54cd09d0a"} Oct 30 05:25:09.029035 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.029011 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:09.196475 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.196443 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-kube-api-access-6j7jk\" (UniqueName: \"kubernetes.io/projected/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-kube-api-access-6j7jk\") pod \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\" (UID: \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\") " Oct 30 05:25:09.196676 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.196539 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-openshift-ca\") pod \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\" (UID: \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\") " Oct 30 05:25:09.196676 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.196596 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-xks-ca\") pod \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\" (UID: \"5fca5e27-9fda-45d7-83f7-fd0d25147bd1\") " Oct 30 05:25:09.196890 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.196866 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-openshift-ca" pod "5fca5e27-9fda-45d7-83f7-fd0d25147bd1" (UID: "5fca5e27-9fda-45d7-83f7-fd0d25147bd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:09.198612 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.198589 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-kube-api-access-6j7jk" pod "5fca5e27-9fda-45d7-83f7-fd0d25147bd1" (UID: "5fca5e27-9fda-45d7-83f7-fd0d25147bd1"). InnerVolumeSpecName "kube-api-access-6j7jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:09.198713 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.198687 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-xks-ca" pod "5fca5e27-9fda-45d7-83f7-fd0d25147bd1" (UID: "5fca5e27-9fda-45d7-83f7-fd0d25147bd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:25:09.297700 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.297673 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j7jk\" (UniqueName: \"kubernetes.io/projected/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-kube-api-access-6j7jk\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:09.297700 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.297695 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:09.297845 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.297705 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fca5e27-9fda-45d7-83f7-fd0d25147bd1-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:09.896352 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.896320 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" event={"ID":"5fca5e27-9fda-45d7-83f7-fd0d25147bd1","Type":"ContainerDied","Data":"432c9170bc41163f4a4023a69d7fd4574cb0337b6292e0192062f2ccb95f024f"} Oct 30 05:25:09.896352 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.896350 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b" Oct 30 05:25:09.896603 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:09.896363 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="432c9170bc41163f4a4023a69d7fd4574cb0337b6292e0192062f2ccb95f024f" Oct 30 05:25:11.217523 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:25:11.217483 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:25:16.647544 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:16.647505 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-bpnfm"] Oct 30 05:25:16.842852 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:16.842816 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:25:17.075690 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:17.075648 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:25:17.296909 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:17.296874 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:25:21.648406 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.648357 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" podUID="951ef5dd-a558-4b58-b22c-b7ffcf625d1c" containerName="payload-processing" containerID="cri-o://9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f" gracePeriod=25 Oct 30 05:25:21.896649 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.896625 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:25:21.962431 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.961803 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-routing-overlay-volume\") pod \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " Oct 30 05:25:21.962431 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.961849 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-plugins-config-volume\") pod \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " Oct 30 05:25:21.962431 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.961938 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-kube-api-access-vl9x5\" (UniqueName: \"kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-kube-api-access-vl9x5\") pod \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " Oct 30 05:25:21.962431 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.961962 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-provider-credentials\") pod \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\" (UID: \"951ef5dd-a558-4b58-b22c-b7ffcf625d1c\") " Oct 30 05:25:21.962431 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.962170 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-routing-overlay-volume" pod "951ef5dd-a558-4b58-b22c-b7ffcf625d1c" (UID: "951ef5dd-a558-4b58-b22c-b7ffcf625d1c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:21.962431 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.962243 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-plugins-config-volume" pod "951ef5dd-a558-4b58-b22c-b7ffcf625d1c" (UID: "951ef5dd-a558-4b58-b22c-b7ffcf625d1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:21.962431 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.962304 4793 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-routing-overlay-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:21.963564 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.963539 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" Oct 30 05:25:21.963860 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.963579 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" event={"ID":"951ef5dd-a558-4b58-b22c-b7ffcf625d1c","Type":"ContainerDied","Data":"9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f"} Oct 30 05:25:21.963997 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.963897 4793 scope.go:122] "RemoveContainer" containerID="9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f" Oct 30 05:25:21.963997 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.963404 4793 generic.go:356] "Generic (PLEG): container finished" podID="951ef5dd-a558-4b58-b22c-b7ffcf625d1c" containerID="9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f" exitCode=0 Oct 30 05:25:21.963997 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.963973 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-758758b45d-bpnfm" event={"ID":"951ef5dd-a558-4b58-b22c-b7ffcf625d1c","Type":"ContainerDied","Data":"eb42e8dcc24bc5f235b5aee7326b99b762a0d9c381823cb69df9d9d4ade9159e"} Oct 30 05:25:21.964490 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.964467 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-provider-credentials" pod "951ef5dd-a558-4b58-b22c-b7ffcf625d1c" (UID: "951ef5dd-a558-4b58-b22c-b7ffcf625d1c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:21.964573 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.964555 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-kube-api-access-vl9x5" pod "951ef5dd-a558-4b58-b22c-b7ffcf625d1c" (UID: "951ef5dd-a558-4b58-b22c-b7ffcf625d1c"). InnerVolumeSpecName "kube-api-access-vl9x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:21.981437 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.981393 4793 scope.go:122] "RemoveContainer" containerID="9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f" Oct 30 05:25:21.981674 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:25:21.981657 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f\": container with ID starting with 9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f not found: ID does not exist" containerID="9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f" Oct 30 05:25:21.981730 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:21.981680 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f"} err="failed to get container status \"9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f\": rpc error: code = NotFound desc = could not find container \"9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f\": container with ID starting with 9b55d868a61f2e6053befcf5aa83f0b85562bc5a1d1d14b9a045e5e4c300e47f not found: ID does not exist" Oct 30 05:25:22.063551 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:22.063520 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:22.063551 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:22.063547 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl9x5\" (UniqueName: \"kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-kube-api-access-vl9x5\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:22.063551 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:22.063557 4793 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/951ef5dd-a558-4b58-b22c-b7ffcf625d1c-provider-credentials\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:22.309056 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:22.309024 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-bpnfm"] Oct 30 05:25:22.312751 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:22.312725 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-758758b45d-bpnfm"] Oct 30 05:25:22.845714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:22.845683 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951ef5dd-a558-4b58-b22c-b7ffcf625d1c" path="/var/lib/kubelet/pods/951ef5dd-a558-4b58-b22c-b7ffcf625d1c/volumes" Oct 30 05:25:29.773491 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.773449 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc"] Oct 30 05:25:29.774103 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.774081 4793 cpu_manager.go:414] "containerMap: removing container" podUID="951ef5dd-a558-4b58-b22c-b7ffcf625d1c" containerName="payload-processing" Oct 30 05:25:29.774145 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.774106 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951ef5dd-a558-4b58-b22c-b7ffcf625d1c" containerName="payload-processing" Oct 30 05:25:29.774182 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.774155 4793 cpu_manager.go:414] "containerMap: removing container" podUID="5fca5e27-9fda-45d7-83f7-fd0d25147bd1" containerName="revoke-keys" Oct 30 05:25:29.774182 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.774164 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fca5e27-9fda-45d7-83f7-fd0d25147bd1" containerName="revoke-keys" Oct 30 05:25:29.774260 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.774249 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="951ef5dd-a558-4b58-b22c-b7ffcf625d1c" containerName="payload-processing" Oct 30 05:25:29.774311 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.774271 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fca5e27-9fda-45d7-83f7-fd0d25147bd1" containerName="revoke-keys" Oct 30 05:25:29.779583 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.779551 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:29.793973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.793948 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:25:29.794486 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.794465 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:25:29.796162 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.796131 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc"] Oct 30 05:25:29.804585 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.804564 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:25:29.804716 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.804587 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:25:29.862934 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.862897 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-rspsc\" (UID: \"aafded1c-d4cb-4209-ba55-52a4f55a0000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:29.863096 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.862957 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-rspsc\" (UID: \"aafded1c-d4cb-4209-ba55-52a4f55a0000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:29.863144 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.863082 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng9kt\" (UniqueName: \"kubernetes.io/projected/aafded1c-d4cb-4209-ba55-52a4f55a0000-kube-api-access-ng9kt\") pod \"maas-api-revoke-sub-c1e271a9-rspsc\" (UID: \"aafded1c-d4cb-4209-ba55-52a4f55a0000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:29.964060 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.964022 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-rspsc\" (UID: \"aafded1c-d4cb-4209-ba55-52a4f55a0000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:29.964216 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.964065 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-rspsc\" (UID: \"aafded1c-d4cb-4209-ba55-52a4f55a0000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:29.964216 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.964106 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng9kt\" (UniqueName: \"kubernetes.io/projected/aafded1c-d4cb-4209-ba55-52a4f55a0000-kube-api-access-ng9kt\") pod \"maas-api-revoke-sub-c1e271a9-rspsc\" (UID: \"aafded1c-d4cb-4209-ba55-52a4f55a0000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:29.964786 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.964763 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-rspsc\" (UID: \"aafded1c-d4cb-4209-ba55-52a4f55a0000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:29.966324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.966301 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-rspsc\" (UID: \"aafded1c-d4cb-4209-ba55-52a4f55a0000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:29.987674 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:29.987653 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng9kt\" (UniqueName: \"kubernetes.io/projected/aafded1c-d4cb-4209-ba55-52a4f55a0000-kube-api-access-ng9kt\") pod \"maas-api-revoke-sub-c1e271a9-rspsc\" (UID: \"aafded1c-d4cb-4209-ba55-52a4f55a0000\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:30.095087 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:30.095016 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:30.394005 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:30.393973 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc"] Oct 30 05:25:30.395446 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:25:30.395382 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafded1c_d4cb_4209_ba55_52a4f55a0000.slice/crio-786e3e31d9c34df86ef079da894083628e8b97e0341d3f2d6b1fe015e80082e5 WatchSource:0}: Error finding container 786e3e31d9c34df86ef079da894083628e8b97e0341d3f2d6b1fe015e80082e5: Status 404 returned error can't find the container with id 786e3e31d9c34df86ef079da894083628e8b97e0341d3f2d6b1fe015e80082e5 Oct 30 05:25:31.058516 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:31.058486 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" event={"ID":"aafded1c-d4cb-4209-ba55-52a4f55a0000","Type":"ContainerStarted","Data":"b021553431f8d24e8cfbe4ad823ba72f61a57cd2d9227ffce79c7fb0234044df"} Oct 30 05:25:31.059067 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:31.058540 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" event={"ID":"aafded1c-d4cb-4209-ba55-52a4f55a0000","Type":"ContainerStarted","Data":"786e3e31d9c34df86ef079da894083628e8b97e0341d3f2d6b1fe015e80082e5"} Oct 30 05:25:31.099886 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:31.099844 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" podStartSLOduration=2.099829774 podStartE2EDuration="2.099829774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:31.09787327 +0000 UTC m=+5254.383748045" watchObservedRunningTime="2026-10-30 05:25:31.099829774 +0000 UTC m=+5254.385704575" Oct 30 05:25:32.064723 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:32.064690 4793 generic.go:356] "Generic (PLEG): container finished" podID="aafded1c-d4cb-4209-ba55-52a4f55a0000" containerID="b021553431f8d24e8cfbe4ad823ba72f61a57cd2d9227ffce79c7fb0234044df" exitCode=0 Oct 30 05:25:32.065114 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:32.064770 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" event={"ID":"aafded1c-d4cb-4209-ba55-52a4f55a0000","Type":"ContainerDied","Data":"b021553431f8d24e8cfbe4ad823ba72f61a57cd2d9227ffce79c7fb0234044df"} Oct 30 05:25:33.214313 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:33.214289 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:33.292455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:33.292402 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-openshift-ca\") pod \"aafded1c-d4cb-4209-ba55-52a4f55a0000\" (UID: \"aafded1c-d4cb-4209-ba55-52a4f55a0000\") " Oct 30 05:25:33.292594 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:33.292481 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-xks-ca\") pod \"aafded1c-d4cb-4209-ba55-52a4f55a0000\" (UID: \"aafded1c-d4cb-4209-ba55-52a4f55a0000\") " Oct 30 05:25:33.292594 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:33.292506 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafded1c-d4cb-4209-ba55-52a4f55a0000-kube-api-access-ng9kt\" (UniqueName: \"kubernetes.io/projected/aafded1c-d4cb-4209-ba55-52a4f55a0000-kube-api-access-ng9kt\") pod \"aafded1c-d4cb-4209-ba55-52a4f55a0000\" (UID: \"aafded1c-d4cb-4209-ba55-52a4f55a0000\") " Oct 30 05:25:33.292704 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:33.292683 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-openshift-ca" pod "aafded1c-d4cb-4209-ba55-52a4f55a0000" (UID: "aafded1c-d4cb-4209-ba55-52a4f55a0000"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:33.292818 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:33.292805 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:33.294471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:33.294445 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-xks-ca" pod "aafded1c-d4cb-4209-ba55-52a4f55a0000" (UID: "aafded1c-d4cb-4209-ba55-52a4f55a0000"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:25:33.294632 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:33.294611 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafded1c-d4cb-4209-ba55-52a4f55a0000-kube-api-access-ng9kt" pod "aafded1c-d4cb-4209-ba55-52a4f55a0000" (UID: "aafded1c-d4cb-4209-ba55-52a4f55a0000"). InnerVolumeSpecName "kube-api-access-ng9kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:33.393872 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:33.393817 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aafded1c-d4cb-4209-ba55-52a4f55a0000-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:33.393872 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:33.393841 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng9kt\" (UniqueName: \"kubernetes.io/projected/aafded1c-d4cb-4209-ba55-52a4f55a0000-kube-api-access-ng9kt\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:34.083315 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:34.083277 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" event={"ID":"aafded1c-d4cb-4209-ba55-52a4f55a0000","Type":"ContainerDied","Data":"786e3e31d9c34df86ef079da894083628e8b97e0341d3f2d6b1fe015e80082e5"} Oct 30 05:25:34.083515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:34.083332 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="786e3e31d9c34df86ef079da894083628e8b97e0341d3f2d6b1fe015e80082e5" Oct 30 05:25:34.083515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:34.083298 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc" Oct 30 05:25:41.219714 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:25:41.219610 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:25:48.031274 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:48.031234 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k"] Oct 30 05:25:48.040022 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:48.039949 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7353741d-jhw7k"] Oct 30 05:25:48.845276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:48.845241 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="761a36c0-39fc-4512-b4db-ab8c788b7504" path="/var/lib/kubelet/pods/761a36c0-39fc-4512-b4db-ab8c788b7504/volumes" Oct 30 05:25:50.074922 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.074887 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68"] Oct 30 05:25:50.075309 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.075285 4793 cpu_manager.go:414] "containerMap: removing container" podUID="aafded1c-d4cb-4209-ba55-52a4f55a0000" containerName="revoke-keys" Oct 30 05:25:50.075309 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.075296 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafded1c-d4cb-4209-ba55-52a4f55a0000" containerName="revoke-keys" Oct 30 05:25:50.075391 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.075366 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafded1c-d4cb-4209-ba55-52a4f55a0000" containerName="revoke-keys" Oct 30 05:25:50.078596 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.078576 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:50.094302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.094273 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:25:50.106064 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.106035 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:25:50.106205 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.106096 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:25:50.106205 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.106137 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:25:50.108396 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.108375 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68"] Oct 30 05:25:50.169267 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.169240 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ghqq\" (UniqueName: \"kubernetes.io/projected/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-kube-api-access-8ghqq\") pod \"maas-api-revoke-sub-2e1f5b5f-lln68\" (UID: \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:50.169409 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.169294 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lln68\" (UID: \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:50.169409 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.169361 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lln68\" (UID: \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:50.270635 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.270603 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ghqq\" (UniqueName: \"kubernetes.io/projected/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-kube-api-access-8ghqq\") pod \"maas-api-revoke-sub-2e1f5b5f-lln68\" (UID: \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:50.270831 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.270666 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lln68\" (UID: \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:50.270831 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.270710 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lln68\" (UID: \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:50.271401 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.271375 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lln68\" (UID: \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:50.272990 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.272966 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-lln68\" (UID: \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:50.293621 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.293594 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ghqq\" (UniqueName: \"kubernetes.io/projected/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-kube-api-access-8ghqq\") pod \"maas-api-revoke-sub-2e1f5b5f-lln68\" (UID: \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:50.391884 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.391797 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:50.693086 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:50.693059 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68"] Oct 30 05:25:50.694537 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:25:50.694512 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87c4fbca_0f9f_40a8_adfb_c59e6f8ac7e5.slice/crio-c55eb561b7ae6b2fafce31f794cf0bf2381b42a21ede43fc421d0a45cd9dd61d WatchSource:0}: Error finding container c55eb561b7ae6b2fafce31f794cf0bf2381b42a21ede43fc421d0a45cd9dd61d: Status 404 returned error can't find the container with id c55eb561b7ae6b2fafce31f794cf0bf2381b42a21ede43fc421d0a45cd9dd61d Oct 30 05:25:51.202234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:51.202204 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" event={"ID":"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5","Type":"ContainerStarted","Data":"110035b61b488c0bab0263ce5c0db3217ab9fd2b349aea3003baf9224f2c5ae6"} Oct 30 05:25:51.202603 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:51.202253 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" event={"ID":"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5","Type":"ContainerStarted","Data":"c55eb561b7ae6b2fafce31f794cf0bf2381b42a21ede43fc421d0a45cd9dd61d"} Oct 30 05:25:51.241819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:51.241766 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" podStartSLOduration=1.241751682 podStartE2EDuration="1.241751682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:25:51.24137827 +0000 UTC m=+5274.527253046" watchObservedRunningTime="2026-10-30 05:25:51.241751682 +0000 UTC m=+5274.527626460" Oct 30 05:25:52.208379 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:52.208343 4793 generic.go:356] "Generic (PLEG): container finished" podID="87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5" containerID="110035b61b488c0bab0263ce5c0db3217ab9fd2b349aea3003baf9224f2c5ae6" exitCode=0 Oct 30 05:25:52.208803 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:52.208385 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" event={"ID":"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5","Type":"ContainerDied","Data":"110035b61b488c0bab0263ce5c0db3217ab9fd2b349aea3003baf9224f2c5ae6"} Oct 30 05:25:53.337192 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:53.337163 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:25:53.353231 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:53.353209 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:53.502748 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:53.502674 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-openshift-ca\") pod \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\" (UID: \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\") " Oct 30 05:25:53.502748 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:53.502745 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-kube-api-access-8ghqq\" (UniqueName: \"kubernetes.io/projected/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-kube-api-access-8ghqq\") pod \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\" (UID: \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\") " Oct 30 05:25:53.502935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:53.502848 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-xks-ca\") pod \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\" (UID: \"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5\") " Oct 30 05:25:53.503020 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:53.502997 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-openshift-ca" pod "87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5" (UID: "87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:25:53.503142 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:53.503126 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:53.504882 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:53.504852 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-kube-api-access-8ghqq" pod "87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5" (UID: "87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5"). InnerVolumeSpecName "kube-api-access-8ghqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:25:53.504980 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:53.504944 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-xks-ca" pod "87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5" (UID: "87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:25:53.547005 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:53.546976 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:25:53.604243 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:53.604216 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ghqq\" (UniqueName: \"kubernetes.io/projected/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-kube-api-access-8ghqq\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:53.604243 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:53.604239 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:25:54.226571 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:54.226539 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" event={"ID":"87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5","Type":"ContainerDied","Data":"c55eb561b7ae6b2fafce31f794cf0bf2381b42a21ede43fc421d0a45cd9dd61d"} Oct 30 05:25:54.226571 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:54.226573 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68" Oct 30 05:25:54.226788 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:25:54.226589 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c55eb561b7ae6b2fafce31f794cf0bf2381b42a21ede43fc421d0a45cd9dd61d" Oct 30 05:26:00.958798 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:00.958758 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:26:01.485208 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.485166 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g"] Oct 30 05:26:01.485714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.485686 4793 cpu_manager.go:414] "containerMap: removing container" podUID="87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5" containerName="revoke-keys" Oct 30 05:26:01.485714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.485712 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5" containerName="revoke-keys" Oct 30 05:26:01.485915 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.485886 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5" containerName="revoke-keys" Oct 30 05:26:01.490684 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.490662 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:01.507736 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.507711 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g"] Oct 30 05:26:01.516540 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.516518 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:26:01.519583 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.519562 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:26:01.519583 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.519574 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:26:01.519781 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.519614 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:26:01.678679 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.678644 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-44n7g\" (UID: \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:01.678854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.678773 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-44n7g\" (UID: \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:01.678854 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.678810 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpsg4\" (UniqueName: \"kubernetes.io/projected/43fddb59-b066-45d5-8cd3-a516d90b8ee2-kube-api-access-gpsg4\") pod \"maas-api-revoke-sub-ec843018-44n7g\" (UID: \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:01.779987 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.779911 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-44n7g\" (UID: \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:01.779987 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.779965 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-44n7g\" (UID: \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:01.780337 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.779989 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpsg4\" (UniqueName: \"kubernetes.io/projected/43fddb59-b066-45d5-8cd3-a516d90b8ee2-kube-api-access-gpsg4\") pod \"maas-api-revoke-sub-ec843018-44n7g\" (UID: \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:01.780774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.780748 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-44n7g\" (UID: \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:01.782472 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.782452 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-44n7g\" (UID: \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:01.803199 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.803173 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpsg4\" (UniqueName: \"kubernetes.io/projected/43fddb59-b066-45d5-8cd3-a516d90b8ee2-kube-api-access-gpsg4\") pod \"maas-api-revoke-sub-ec843018-44n7g\" (UID: \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:01.803906 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:01.803888 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:02.139410 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:02.139385 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g"] Oct 30 05:26:02.141558 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:26:02.141527 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fddb59_b066_45d5_8cd3_a516d90b8ee2.slice/crio-3dddf42877bb8d7d847bd39b8d287595cf924cf38027ebc4126bd97a150e1223 WatchSource:0}: Error finding container 3dddf42877bb8d7d847bd39b8d287595cf924cf38027ebc4126bd97a150e1223: Status 404 returned error can't find the container with id 3dddf42877bb8d7d847bd39b8d287595cf924cf38027ebc4126bd97a150e1223 Oct 30 05:26:02.270943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:02.270915 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" event={"ID":"43fddb59-b066-45d5-8cd3-a516d90b8ee2","Type":"ContainerStarted","Data":"3dddf42877bb8d7d847bd39b8d287595cf924cf38027ebc4126bd97a150e1223"} Oct 30 05:26:03.277459 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:03.277398 4793 generic.go:356] "Generic (PLEG): container finished" podID="43fddb59-b066-45d5-8cd3-a516d90b8ee2" containerID="cc1d8870e9383687c31c4f19b3c148841f088fc696d8b6b9385b6df9c490814d" exitCode=0 Oct 30 05:26:03.277908 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:03.277483 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" event={"ID":"43fddb59-b066-45d5-8cd3-a516d90b8ee2","Type":"ContainerDied","Data":"cc1d8870e9383687c31c4f19b3c148841f088fc696d8b6b9385b6df9c490814d"} Oct 30 05:26:04.419359 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:04.419334 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:04.506664 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:04.506632 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-xks-ca\") pod \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\" (UID: \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\") " Oct 30 05:26:04.506835 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:04.506726 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fddb59-b066-45d5-8cd3-a516d90b8ee2-kube-api-access-gpsg4\" (UniqueName: \"kubernetes.io/projected/43fddb59-b066-45d5-8cd3-a516d90b8ee2-kube-api-access-gpsg4\") pod \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\" (UID: \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\") " Oct 30 05:26:04.506917 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:04.506871 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-openshift-ca\") pod \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\" (UID: \"43fddb59-b066-45d5-8cd3-a516d90b8ee2\") " Oct 30 05:26:04.507191 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:04.507169 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-openshift-ca" pod "43fddb59-b066-45d5-8cd3-a516d90b8ee2" (UID: "43fddb59-b066-45d5-8cd3-a516d90b8ee2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:26:04.508790 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:04.508765 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fddb59-b066-45d5-8cd3-a516d90b8ee2-kube-api-access-gpsg4" pod "43fddb59-b066-45d5-8cd3-a516d90b8ee2" (UID: "43fddb59-b066-45d5-8cd3-a516d90b8ee2"). InnerVolumeSpecName "kube-api-access-gpsg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:04.509031 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:04.509011 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-xks-ca" pod "43fddb59-b066-45d5-8cd3-a516d90b8ee2" (UID: "43fddb59-b066-45d5-8cd3-a516d90b8ee2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:26:04.607734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:04.607650 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:26:04.607734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:04.607686 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43fddb59-b066-45d5-8cd3-a516d90b8ee2-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:26:04.607734 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:04.607701 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpsg4\" (UniqueName: \"kubernetes.io/projected/43fddb59-b066-45d5-8cd3-a516d90b8ee2-kube-api-access-gpsg4\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:26:05.290092 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:05.290065 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" Oct 30 05:26:05.290292 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:05.290093 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g" event={"ID":"43fddb59-b066-45d5-8cd3-a516d90b8ee2","Type":"ContainerDied","Data":"3dddf42877bb8d7d847bd39b8d287595cf924cf38027ebc4126bd97a150e1223"} Oct 30 05:26:05.290292 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:05.290155 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dddf42877bb8d7d847bd39b8d287595cf924cf38027ebc4126bd97a150e1223" Oct 30 05:26:11.218412 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:26:11.218373 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:26:29.575572 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.575534 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq"] Oct 30 05:26:29.577282 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.577255 4793 cpu_manager.go:414] "containerMap: removing container" podUID="43fddb59-b066-45d5-8cd3-a516d90b8ee2" containerName="revoke-keys" Oct 30 05:26:29.577469 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.577454 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43fddb59-b066-45d5-8cd3-a516d90b8ee2" containerName="revoke-keys" Oct 30 05:26:29.577792 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.577777 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="43fddb59-b066-45d5-8cd3-a516d90b8ee2" containerName="revoke-keys" Oct 30 05:26:29.582059 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.582026 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:29.595468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.595433 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq"] Oct 30 05:26:29.603983 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.603957 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:26:29.604098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.604057 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:26:29.608227 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.608204 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:26:29.608227 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.608224 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:26:29.629045 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.629011 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmswm\" (UniqueName: \"kubernetes.io/projected/4dfebbac-4dcf-424f-b612-7a152d071d74-kube-api-access-vmswm\") pod \"maas-api-revoke-sub-046e1343-kl5mq\" (UID: \"4dfebbac-4dcf-424f-b612-7a152d071d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:29.629178 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.629055 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-kl5mq\" (UID: \"4dfebbac-4dcf-424f-b612-7a152d071d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:29.629178 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.629090 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-kl5mq\" (UID: \"4dfebbac-4dcf-424f-b612-7a152d071d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:29.730553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.730514 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-kl5mq\" (UID: \"4dfebbac-4dcf-424f-b612-7a152d071d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:29.730713 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.730623 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmswm\" (UniqueName: \"kubernetes.io/projected/4dfebbac-4dcf-424f-b612-7a152d071d74-kube-api-access-vmswm\") pod \"maas-api-revoke-sub-046e1343-kl5mq\" (UID: \"4dfebbac-4dcf-424f-b612-7a152d071d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:29.730713 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.730646 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-kl5mq\" (UID: \"4dfebbac-4dcf-424f-b612-7a152d071d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:29.731232 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.731211 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-kl5mq\" (UID: \"4dfebbac-4dcf-424f-b612-7a152d071d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:29.732762 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.732740 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-kl5mq\" (UID: \"4dfebbac-4dcf-424f-b612-7a152d071d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:29.753814 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.753794 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmswm\" (UniqueName: \"kubernetes.io/projected/4dfebbac-4dcf-424f-b612-7a152d071d74-kube-api-access-vmswm\") pod \"maas-api-revoke-sub-046e1343-kl5mq\" (UID: \"4dfebbac-4dcf-424f-b612-7a152d071d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:29.896732 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:29.896671 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:30.191444 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:30.191394 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq"] Oct 30 05:26:30.193272 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:26:30.193229 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfebbac_4dcf_424f_b612_7a152d071d74.slice/crio-644c431507845a1605d46aee2fa2ab872821f76d7353994abf78ede710fe2028 WatchSource:0}: Error finding container 644c431507845a1605d46aee2fa2ab872821f76d7353994abf78ede710fe2028: Status 404 returned error can't find the container with id 644c431507845a1605d46aee2fa2ab872821f76d7353994abf78ede710fe2028 Oct 30 05:26:30.442635 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:30.442575 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" event={"ID":"4dfebbac-4dcf-424f-b612-7a152d071d74","Type":"ContainerStarted","Data":"644c431507845a1605d46aee2fa2ab872821f76d7353994abf78ede710fe2028"} Oct 30 05:26:31.449334 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:31.449299 4793 generic.go:356] "Generic (PLEG): container finished" podID="4dfebbac-4dcf-424f-b612-7a152d071d74" containerID="6f125db57c9fdb6bf0812b7ea4b30e607ab46ad71d8e07fb621bd4ea68276578" exitCode=0 Oct 30 05:26:31.449731 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:31.449380 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" event={"ID":"4dfebbac-4dcf-424f-b612-7a152d071d74","Type":"ContainerDied","Data":"6f125db57c9fdb6bf0812b7ea4b30e607ab46ad71d8e07fb621bd4ea68276578"} Oct 30 05:26:32.586698 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:32.586678 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:32.654080 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:32.654055 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dfebbac-4dcf-424f-b612-7a152d071d74-kube-api-access-vmswm\" (UniqueName: \"kubernetes.io/projected/4dfebbac-4dcf-424f-b612-7a152d071d74-kube-api-access-vmswm\") pod \"4dfebbac-4dcf-424f-b612-7a152d071d74\" (UID: \"4dfebbac-4dcf-424f-b612-7a152d071d74\") " Oct 30 05:26:32.654288 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:32.654268 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-xks-ca\") pod \"4dfebbac-4dcf-424f-b612-7a152d071d74\" (UID: \"4dfebbac-4dcf-424f-b612-7a152d071d74\") " Oct 30 05:26:32.654338 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:32.654323 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-openshift-ca\") pod \"4dfebbac-4dcf-424f-b612-7a152d071d74\" (UID: \"4dfebbac-4dcf-424f-b612-7a152d071d74\") " Oct 30 05:26:32.654712 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:32.654688 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-openshift-ca" pod "4dfebbac-4dcf-424f-b612-7a152d071d74" (UID: "4dfebbac-4dcf-424f-b612-7a152d071d74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:26:32.656158 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:32.656130 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dfebbac-4dcf-424f-b612-7a152d071d74-kube-api-access-vmswm" pod "4dfebbac-4dcf-424f-b612-7a152d071d74" (UID: "4dfebbac-4dcf-424f-b612-7a152d071d74"). InnerVolumeSpecName "kube-api-access-vmswm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:32.656272 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:32.656214 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-xks-ca" pod "4dfebbac-4dcf-424f-b612-7a152d071d74" (UID: "4dfebbac-4dcf-424f-b612-7a152d071d74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:26:32.755866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:32.755814 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmswm\" (UniqueName: \"kubernetes.io/projected/4dfebbac-4dcf-424f-b612-7a152d071d74-kube-api-access-vmswm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:26:32.755866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:32.755838 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:26:32.755866 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:32.755847 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dfebbac-4dcf-424f-b612-7a152d071d74-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:26:33.463576 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:33.463545 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" Oct 30 05:26:33.463753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:33.463580 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq" event={"ID":"4dfebbac-4dcf-424f-b612-7a152d071d74","Type":"ContainerDied","Data":"644c431507845a1605d46aee2fa2ab872821f76d7353994abf78ede710fe2028"} Oct 30 05:26:33.463753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:33.463625 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="644c431507845a1605d46aee2fa2ab872821f76d7353994abf78ede710fe2028" Oct 30 05:26:39.987341 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:39.987305 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv"] Oct 30 05:26:39.987952 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:39.987735 4793 cpu_manager.go:414] "containerMap: removing container" podUID="4dfebbac-4dcf-424f-b612-7a152d071d74" containerName="revoke-keys" Oct 30 05:26:39.987952 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:39.987748 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dfebbac-4dcf-424f-b612-7a152d071d74" containerName="revoke-keys" Oct 30 05:26:39.987952 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:39.987813 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dfebbac-4dcf-424f-b612-7a152d071d74" containerName="revoke-keys" Oct 30 05:26:39.991068 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:39.991048 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:40.006341 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.006310 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:26:40.018455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.018398 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:26:40.018566 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.018402 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:26:40.018566 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.018402 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:26:40.020979 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.020959 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv"] Oct 30 05:26:40.117990 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.117956 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwc4l\" (UniqueName: \"kubernetes.io/projected/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-kube-api-access-rwc4l\") pod \"maas-api-revoke-sub-d3fba056-652lv\" (UID: \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:40.118155 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.118000 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-652lv\" (UID: \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:40.118155 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.118142 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-652lv\" (UID: \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:40.219346 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.219315 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwc4l\" (UniqueName: \"kubernetes.io/projected/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-kube-api-access-rwc4l\") pod \"maas-api-revoke-sub-d3fba056-652lv\" (UID: \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:40.219557 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.219360 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-652lv\" (UID: \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:40.219557 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.219456 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-652lv\" (UID: \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:40.220185 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.220159 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-652lv\" (UID: \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:40.221743 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.221720 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-652lv\" (UID: \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:40.242305 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.242251 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwc4l\" (UniqueName: \"kubernetes.io/projected/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-kube-api-access-rwc4l\") pod \"maas-api-revoke-sub-d3fba056-652lv\" (UID: \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:40.303685 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.303655 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:40.601163 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:40.601139 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv"] Oct 30 05:26:40.602840 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:26:40.602808 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9dd94e_0cf7_47c9_b0d5_6946929d91a1.slice/crio-b94d2b73ccf1073b91e62bab42e10c8a18704b13778f83f0184166b4f4d521a3 WatchSource:0}: Error finding container b94d2b73ccf1073b91e62bab42e10c8a18704b13778f83f0184166b4f4d521a3: Status 404 returned error can't find the container with id b94d2b73ccf1073b91e62bab42e10c8a18704b13778f83f0184166b4f4d521a3 Oct 30 05:26:41.219370 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:26:41.219335 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:26:41.511324 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:41.511235 4793 generic.go:356] "Generic (PLEG): container finished" podID="1d9dd94e-0cf7-47c9-b0d5-6946929d91a1" containerID="99f3c7ebf74accee8b0c3e62061c58b7050ea034663c1d9f2fe0f24b9c2ae9fa" exitCode=0 Oct 30 05:26:41.511510 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:41.511373 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" event={"ID":"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1","Type":"ContainerDied","Data":"99f3c7ebf74accee8b0c3e62061c58b7050ea034663c1d9f2fe0f24b9c2ae9fa"} Oct 30 05:26:41.511510 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:41.511470 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" event={"ID":"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1","Type":"ContainerStarted","Data":"b94d2b73ccf1073b91e62bab42e10c8a18704b13778f83f0184166b4f4d521a3"} Oct 30 05:26:42.962881 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:42.962854 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:43.064441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:43.052288 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-xks-ca\") pod \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\" (UID: \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\") " Oct 30 05:26:43.064441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:43.052350 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-openshift-ca\") pod \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\" (UID: \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\") " Oct 30 05:26:43.064441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:43.052395 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-kube-api-access-rwc4l\" (UniqueName: \"kubernetes.io/projected/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-kube-api-access-rwc4l\") pod \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\" (UID: \"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1\") " Oct 30 05:26:43.064441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:43.053229 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-openshift-ca" pod "1d9dd94e-0cf7-47c9-b0d5-6946929d91a1" (UID: "1d9dd94e-0cf7-47c9-b0d5-6946929d91a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:26:43.090458 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:43.065553 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-kube-api-access-rwc4l" pod "1d9dd94e-0cf7-47c9-b0d5-6946929d91a1" (UID: "1d9dd94e-0cf7-47c9-b0d5-6946929d91a1"). InnerVolumeSpecName "kube-api-access-rwc4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:26:43.109915 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:43.109883 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-xks-ca" pod "1d9dd94e-0cf7-47c9-b0d5-6946929d91a1" (UID: "1d9dd94e-0cf7-47c9-b0d5-6946929d91a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:26:43.153387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:43.153322 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:26:43.153387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:43.153349 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:26:43.153387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:43.153359 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwc4l\" (UniqueName: \"kubernetes.io/projected/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1-kube-api-access-rwc4l\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:26:43.542531 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:43.542494 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" Oct 30 05:26:43.542755 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:43.542575 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv" event={"ID":"1d9dd94e-0cf7-47c9-b0d5-6946929d91a1","Type":"ContainerDied","Data":"b94d2b73ccf1073b91e62bab42e10c8a18704b13778f83f0184166b4f4d521a3"} Oct 30 05:26:43.542755 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:43.542632 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b94d2b73ccf1073b91e62bab42e10c8a18704b13778f83f0184166b4f4d521a3" Oct 30 05:26:54.973135 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:26:54.973098 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:27:00.138102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:00.138066 4793 scope.go:122] "RemoveContainer" containerID="2515b285c79b4f274dba3c47881145b24c82b936137a3770f04098e3cf39d4af" Oct 30 05:27:01.086038 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.086004 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:27:01.622146 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.622116 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d"] Oct 30 05:27:01.622557 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.622529 4793 cpu_manager.go:414] "containerMap: removing container" podUID="1d9dd94e-0cf7-47c9-b0d5-6946929d91a1" containerName="revoke-keys" Oct 30 05:27:01.622557 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.622542 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9dd94e-0cf7-47c9-b0d5-6946929d91a1" containerName="revoke-keys" Oct 30 05:27:01.622634 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.622621 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9dd94e-0cf7-47c9-b0d5-6946929d91a1" containerName="revoke-keys" Oct 30 05:27:01.627442 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.627400 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:01.644958 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.644934 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d"] Oct 30 05:27:01.654367 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.654342 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:27:01.654367 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.654369 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:27:01.654553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.654371 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:27:01.654553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.654469 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:27:01.772732 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.772683 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-f5g5d\" (UID: \"260b9450-132e-42ea-a1a4-987c65627c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:01.772732 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.772741 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-f5g5d\" (UID: \"260b9450-132e-42ea-a1a4-987c65627c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:01.772992 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.772842 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfm9m\" (UniqueName: \"kubernetes.io/projected/260b9450-132e-42ea-a1a4-987c65627c08-kube-api-access-nfm9m\") pod \"maas-api-revoke-sub-ae594979-f5g5d\" (UID: \"260b9450-132e-42ea-a1a4-987c65627c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:01.873436 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.873327 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-f5g5d\" (UID: \"260b9450-132e-42ea-a1a4-987c65627c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:01.873436 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.873380 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-f5g5d\" (UID: \"260b9450-132e-42ea-a1a4-987c65627c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:01.873659 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.873456 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfm9m\" (UniqueName: \"kubernetes.io/projected/260b9450-132e-42ea-a1a4-987c65627c08-kube-api-access-nfm9m\") pod \"maas-api-revoke-sub-ae594979-f5g5d\" (UID: \"260b9450-132e-42ea-a1a4-987c65627c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:01.874054 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.874026 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-f5g5d\" (UID: \"260b9450-132e-42ea-a1a4-987c65627c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:01.875638 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.875619 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-f5g5d\" (UID: \"260b9450-132e-42ea-a1a4-987c65627c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:01.896781 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.896754 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfm9m\" (UniqueName: \"kubernetes.io/projected/260b9450-132e-42ea-a1a4-987c65627c08-kube-api-access-nfm9m\") pod \"maas-api-revoke-sub-ae594979-f5g5d\" (UID: \"260b9450-132e-42ea-a1a4-987c65627c08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:01.940142 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:01.940119 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:02.237771 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:02.237736 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d"] Oct 30 05:27:02.239515 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:27:02.239487 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260b9450_132e_42ea_a1a4_987c65627c08.slice/crio-85084278c3a9732596516ab7f28865490d94d3e8823fe994eb840b2e04804b96 WatchSource:0}: Error finding container 85084278c3a9732596516ab7f28865490d94d3e8823fe994eb840b2e04804b96: Status 404 returned error can't find the container with id 85084278c3a9732596516ab7f28865490d94d3e8823fe994eb840b2e04804b96 Oct 30 05:27:02.659589 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:02.659549 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" event={"ID":"260b9450-132e-42ea-a1a4-987c65627c08","Type":"ContainerStarted","Data":"1e55df745334cc13872eda33e143fbd0d3ea43438abd4d3e55efb731c1497b6b"} Oct 30 05:27:02.659981 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:02.659613 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" event={"ID":"260b9450-132e-42ea-a1a4-987c65627c08","Type":"ContainerStarted","Data":"85084278c3a9732596516ab7f28865490d94d3e8823fe994eb840b2e04804b96"} Oct 30 05:27:02.701095 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:02.701047 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" podStartSLOduration=1.701033653 podStartE2EDuration="1.701033653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:02.699012849 +0000 UTC m=+5345.984887646" watchObservedRunningTime="2026-10-30 05:27:02.701033653 +0000 UTC m=+5345.986908428" Oct 30 05:27:03.666754 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:03.666718 4793 generic.go:356] "Generic (PLEG): container finished" podID="260b9450-132e-42ea-a1a4-987c65627c08" containerID="1e55df745334cc13872eda33e143fbd0d3ea43438abd4d3e55efb731c1497b6b" exitCode=0 Oct 30 05:27:03.667152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:03.666827 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" event={"ID":"260b9450-132e-42ea-a1a4-987c65627c08","Type":"ContainerDied","Data":"1e55df745334cc13872eda33e143fbd0d3ea43438abd4d3e55efb731c1497b6b"} Oct 30 05:27:04.799236 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:04.799207 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:04.896471 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:04.896439 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260b9450-132e-42ea-a1a4-987c65627c08-kube-api-access-nfm9m\" (UniqueName: \"kubernetes.io/projected/260b9450-132e-42ea-a1a4-987c65627c08-kube-api-access-nfm9m\") pod \"260b9450-132e-42ea-a1a4-987c65627c08\" (UID: \"260b9450-132e-42ea-a1a4-987c65627c08\") " Oct 30 05:27:04.896654 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:04.896547 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-openshift-ca\") pod \"260b9450-132e-42ea-a1a4-987c65627c08\" (UID: \"260b9450-132e-42ea-a1a4-987c65627c08\") " Oct 30 05:27:04.896654 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:04.896606 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-xks-ca\") pod \"260b9450-132e-42ea-a1a4-987c65627c08\" (UID: \"260b9450-132e-42ea-a1a4-987c65627c08\") " Oct 30 05:27:04.896888 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:04.896867 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-openshift-ca" pod "260b9450-132e-42ea-a1a4-987c65627c08" (UID: "260b9450-132e-42ea-a1a4-987c65627c08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:27:04.898488 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:04.898463 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-xks-ca" pod "260b9450-132e-42ea-a1a4-987c65627c08" (UID: "260b9450-132e-42ea-a1a4-987c65627c08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:27:04.898600 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:04.898552 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260b9450-132e-42ea-a1a4-987c65627c08-kube-api-access-nfm9m" pod "260b9450-132e-42ea-a1a4-987c65627c08" (UID: "260b9450-132e-42ea-a1a4-987c65627c08"). InnerVolumeSpecName "kube-api-access-nfm9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:27:04.997330 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:04.997293 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfm9m\" (UniqueName: \"kubernetes.io/projected/260b9450-132e-42ea-a1a4-987c65627c08-kube-api-access-nfm9m\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:27:04.997330 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:04.997329 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:27:04.997330 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:04.997341 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/260b9450-132e-42ea-a1a4-987c65627c08-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:27:05.678698 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:05.678671 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" Oct 30 05:27:05.678956 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:05.678697 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d" event={"ID":"260b9450-132e-42ea-a1a4-987c65627c08","Type":"ContainerDied","Data":"85084278c3a9732596516ab7f28865490d94d3e8823fe994eb840b2e04804b96"} Oct 30 05:27:05.678956 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:05.678742 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85084278c3a9732596516ab7f28865490d94d3e8823fe994eb840b2e04804b96" Oct 30 05:27:11.218488 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:27:11.218386 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:27:16.919935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:16.919891 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt"] Oct 30 05:27:16.921084 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:16.921055 4793 cpu_manager.go:414] "containerMap: removing container" podUID="260b9450-132e-42ea-a1a4-987c65627c08" containerName="revoke-keys" Oct 30 05:27:16.921237 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:16.921221 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260b9450-132e-42ea-a1a4-987c65627c08" containerName="revoke-keys" Oct 30 05:27:16.921498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:16.921480 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="260b9450-132e-42ea-a1a4-987c65627c08" containerName="revoke-keys" Oct 30 05:27:16.925167 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:16.925145 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:16.937456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:16.929873 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt"] Oct 30 05:27:16.940310 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:16.939681 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:27:16.951295 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:16.951273 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:27:16.951394 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:16.951316 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:27:16.951394 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:16.951351 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:27:17.035306 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:17.035264 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqc59\" (UniqueName: \"kubernetes.io/projected/5e93211f-42e5-4c75-a2fe-55f0cb874b47-kube-api-access-mqc59\") pod \"maas-api-revoke-sub-a4aef7d7-74wzt\" (UID: \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:17.035625 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:17.035599 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-74wzt\" (UID: \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:17.036009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:17.035855 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-74wzt\" (UID: \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:17.138990 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:17.138951 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqc59\" (UniqueName: \"kubernetes.io/projected/5e93211f-42e5-4c75-a2fe-55f0cb874b47-kube-api-access-mqc59\") pod \"maas-api-revoke-sub-a4aef7d7-74wzt\" (UID: \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:17.139174 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:17.139009 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-74wzt\" (UID: \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:17.139174 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:17.139072 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-74wzt\" (UID: \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:17.139978 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:17.139948 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-74wzt\" (UID: \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:17.143436 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:17.143398 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-74wzt\" (UID: \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:17.177509 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:17.177488 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqc59\" (UniqueName: \"kubernetes.io/projected/5e93211f-42e5-4c75-a2fe-55f0cb874b47-kube-api-access-mqc59\") pod \"maas-api-revoke-sub-a4aef7d7-74wzt\" (UID: \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:17.253139 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:17.253110 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:17.943551 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:17.943402 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt"] Oct 30 05:27:17.951632 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:27:17.951585 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e93211f_42e5_4c75_a2fe_55f0cb874b47.slice/crio-37517305a462b513241e6835573796b0a1907747815e1ca52ed4d307e00ac99d WatchSource:0}: Error finding container 37517305a462b513241e6835573796b0a1907747815e1ca52ed4d307e00ac99d: Status 404 returned error can't find the container with id 37517305a462b513241e6835573796b0a1907747815e1ca52ed4d307e00ac99d Oct 30 05:27:18.799713 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:18.799664 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" event={"ID":"5e93211f-42e5-4c75-a2fe-55f0cb874b47","Type":"ContainerStarted","Data":"2f8b7433d9b04e9aabfdd9301156a8e38389a85ed6c3be92a209f149cfd0ab08"} Oct 30 05:27:18.799918 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:18.799768 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" event={"ID":"5e93211f-42e5-4c75-a2fe-55f0cb874b47","Type":"ContainerStarted","Data":"37517305a462b513241e6835573796b0a1907747815e1ca52ed4d307e00ac99d"} Oct 30 05:27:18.841838 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:18.841783 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" podStartSLOduration=2.8417698810000003 podStartE2EDuration="2.841769881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:18.838772258 +0000 UTC m=+5362.124647032" watchObservedRunningTime="2026-10-30 05:27:18.841769881 +0000 UTC m=+5362.127644656" Oct 30 05:27:19.805796 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:19.805760 4793 generic.go:356] "Generic (PLEG): container finished" podID="5e93211f-42e5-4c75-a2fe-55f0cb874b47" containerID="2f8b7433d9b04e9aabfdd9301156a8e38389a85ed6c3be92a209f149cfd0ab08" exitCode=0 Oct 30 05:27:19.806234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:19.805847 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" event={"ID":"5e93211f-42e5-4c75-a2fe-55f0cb874b47","Type":"ContainerDied","Data":"2f8b7433d9b04e9aabfdd9301156a8e38389a85ed6c3be92a209f149cfd0ab08"} Oct 30 05:27:20.945638 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:20.945613 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:20.966012 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:20.965986 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-openshift-ca\") pod \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\" (UID: \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\") " Oct 30 05:27:20.966123 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:20.966052 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e93211f-42e5-4c75-a2fe-55f0cb874b47-kube-api-access-mqc59\" (UniqueName: \"kubernetes.io/projected/5e93211f-42e5-4c75-a2fe-55f0cb874b47-kube-api-access-mqc59\") pod \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\" (UID: \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\") " Oct 30 05:27:20.966172 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:20.966149 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-xks-ca\") pod \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\" (UID: \"5e93211f-42e5-4c75-a2fe-55f0cb874b47\") " Oct 30 05:27:20.966382 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:20.966355 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-openshift-ca" pod "5e93211f-42e5-4c75-a2fe-55f0cb874b47" (UID: "5e93211f-42e5-4c75-a2fe-55f0cb874b47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:27:20.966469 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:20.966454 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:27:20.968078 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:20.968058 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e93211f-42e5-4c75-a2fe-55f0cb874b47-kube-api-access-mqc59" pod "5e93211f-42e5-4c75-a2fe-55f0cb874b47" (UID: "5e93211f-42e5-4c75-a2fe-55f0cb874b47"). InnerVolumeSpecName "kube-api-access-mqc59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:27:20.968138 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:20.968056 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-xks-ca" pod "5e93211f-42e5-4c75-a2fe-55f0cb874b47" (UID: "5e93211f-42e5-4c75-a2fe-55f0cb874b47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:27:21.067928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:21.067861 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqc59\" (UniqueName: \"kubernetes.io/projected/5e93211f-42e5-4c75-a2fe-55f0cb874b47-kube-api-access-mqc59\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:27:21.067928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:21.067885 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e93211f-42e5-4c75-a2fe-55f0cb874b47-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:27:21.818840 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:21.818802 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" event={"ID":"5e93211f-42e5-4c75-a2fe-55f0cb874b47","Type":"ContainerDied","Data":"37517305a462b513241e6835573796b0a1907747815e1ca52ed4d307e00ac99d"} Oct 30 05:27:21.818840 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:21.818835 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt" Oct 30 05:27:21.819061 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:21.818861 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37517305a462b513241e6835573796b0a1907747815e1ca52ed4d307e00ac99d" Oct 30 05:27:33.532907 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.532866 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:27:33.876800 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.876717 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb"] Oct 30 05:27:33.877309 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.877287 4793 cpu_manager.go:414] "containerMap: removing container" podUID="5e93211f-42e5-4c75-a2fe-55f0cb874b47" containerName="revoke-keys" Oct 30 05:27:33.877386 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.877313 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e93211f-42e5-4c75-a2fe-55f0cb874b47" containerName="revoke-keys" Oct 30 05:27:33.877479 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.877465 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e93211f-42e5-4c75-a2fe-55f0cb874b47" containerName="revoke-keys" Oct 30 05:27:33.881215 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.881184 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:33.926145 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.926107 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb"] Oct 30 05:27:33.926591 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.926567 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:27:33.943458 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.938393 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:27:33.944804 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.944605 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:27:33.944804 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.944716 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:27:33.985452 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.984644 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jc5kb\" (UID: \"24d7e926-b408-4f0e-b13a-a629ed6a6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:33.985452 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.984780 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9hjv\" (UniqueName: \"kubernetes.io/projected/24d7e926-b408-4f0e-b13a-a629ed6a6421-kube-api-access-h9hjv\") pod \"maas-api-revoke-sub-79805ca9-jc5kb\" (UID: \"24d7e926-b408-4f0e-b13a-a629ed6a6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:33.985452 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:33.984877 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jc5kb\" (UID: \"24d7e926-b408-4f0e-b13a-a629ed6a6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:34.086580 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:34.085932 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jc5kb\" (UID: \"24d7e926-b408-4f0e-b13a-a629ed6a6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:34.086580 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:34.086061 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jc5kb\" (UID: \"24d7e926-b408-4f0e-b13a-a629ed6a6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:34.086580 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:34.086132 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9hjv\" (UniqueName: \"kubernetes.io/projected/24d7e926-b408-4f0e-b13a-a629ed6a6421-kube-api-access-h9hjv\") pod \"maas-api-revoke-sub-79805ca9-jc5kb\" (UID: \"24d7e926-b408-4f0e-b13a-a629ed6a6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:34.087198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:34.087168 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-jc5kb\" (UID: \"24d7e926-b408-4f0e-b13a-a629ed6a6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:34.103579 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:34.091832 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-jc5kb\" (UID: \"24d7e926-b408-4f0e-b13a-a629ed6a6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:34.117144 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:34.117115 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9hjv\" (UniqueName: \"kubernetes.io/projected/24d7e926-b408-4f0e-b13a-a629ed6a6421-kube-api-access-h9hjv\") pod \"maas-api-revoke-sub-79805ca9-jc5kb\" (UID: \"24d7e926-b408-4f0e-b13a-a629ed6a6421\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:34.193440 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:34.190441 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:27:34.259441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:34.257558 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:34.731830 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:34.729581 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb"] Oct 30 05:27:34.733044 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:27:34.732979 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d7e926_b408_4f0e_b13a_a629ed6a6421.slice/crio-3b16835d7703e58f239f8a25fa53e251b9861618aa8934bdac9569602fe41356 WatchSource:0}: Error finding container 3b16835d7703e58f239f8a25fa53e251b9861618aa8934bdac9569602fe41356: Status 404 returned error can't find the container with id 3b16835d7703e58f239f8a25fa53e251b9861618aa8934bdac9569602fe41356 Oct 30 05:27:34.924782 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:34.924734 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" event={"ID":"24d7e926-b408-4f0e-b13a-a629ed6a6421","Type":"ContainerStarted","Data":"3b16835d7703e58f239f8a25fa53e251b9861618aa8934bdac9569602fe41356"} Oct 30 05:27:35.930850 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:35.930820 4793 generic.go:356] "Generic (PLEG): container finished" podID="24d7e926-b408-4f0e-b13a-a629ed6a6421" containerID="b0f848bd6bed36f0afae67a52838177d51496e8c1d123428fe952bc9c7c8a7c5" exitCode=0 Oct 30 05:27:35.931261 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:35.930902 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" event={"ID":"24d7e926-b408-4f0e-b13a-a629ed6a6421","Type":"ContainerDied","Data":"b0f848bd6bed36f0afae67a52838177d51496e8c1d123428fe952bc9c7c8a7c5"} Oct 30 05:27:37.083899 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.083870 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:37.257068 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.256984 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d7e926-b408-4f0e-b13a-a629ed6a6421-kube-api-access-h9hjv\" (UniqueName: \"kubernetes.io/projected/24d7e926-b408-4f0e-b13a-a629ed6a6421-kube-api-access-h9hjv\") pod \"24d7e926-b408-4f0e-b13a-a629ed6a6421\" (UID: \"24d7e926-b408-4f0e-b13a-a629ed6a6421\") " Oct 30 05:27:37.257068 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.257047 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-openshift-ca\") pod \"24d7e926-b408-4f0e-b13a-a629ed6a6421\" (UID: \"24d7e926-b408-4f0e-b13a-a629ed6a6421\") " Oct 30 05:27:37.257313 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.257140 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-xks-ca\") pod \"24d7e926-b408-4f0e-b13a-a629ed6a6421\" (UID: \"24d7e926-b408-4f0e-b13a-a629ed6a6421\") " Oct 30 05:27:37.257382 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.257358 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-openshift-ca" pod "24d7e926-b408-4f0e-b13a-a629ed6a6421" (UID: "24d7e926-b408-4f0e-b13a-a629ed6a6421"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:27:37.259176 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.259145 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d7e926-b408-4f0e-b13a-a629ed6a6421-kube-api-access-h9hjv" pod "24d7e926-b408-4f0e-b13a-a629ed6a6421" (UID: "24d7e926-b408-4f0e-b13a-a629ed6a6421"). InnerVolumeSpecName "kube-api-access-h9hjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:27:37.259302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.259243 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-xks-ca" pod "24d7e926-b408-4f0e-b13a-a629ed6a6421" (UID: "24d7e926-b408-4f0e-b13a-a629ed6a6421"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:27:37.358230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.358197 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9hjv\" (UniqueName: \"kubernetes.io/projected/24d7e926-b408-4f0e-b13a-a629ed6a6421-kube-api-access-h9hjv\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:27:37.358230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.358224 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:27:37.358230 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.358236 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d7e926-b408-4f0e-b13a-a629ed6a6421-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:27:37.942449 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.942405 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" Oct 30 05:27:37.942635 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.942399 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb" event={"ID":"24d7e926-b408-4f0e-b13a-a629ed6a6421","Type":"ContainerDied","Data":"3b16835d7703e58f239f8a25fa53e251b9861618aa8934bdac9569602fe41356"} Oct 30 05:27:37.942635 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:37.942549 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b16835d7703e58f239f8a25fa53e251b9861618aa8934bdac9569602fe41356" Oct 30 05:27:41.218889 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:27:41.218848 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:27:47.578614 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.578570 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9"] Oct 30 05:27:47.579868 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.579839 4793 cpu_manager.go:414] "containerMap: removing container" podUID="24d7e926-b408-4f0e-b13a-a629ed6a6421" containerName="revoke-keys" Oct 30 05:27:47.580037 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.580022 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d7e926-b408-4f0e-b13a-a629ed6a6421" containerName="revoke-keys" Oct 30 05:27:47.580302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.580286 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d7e926-b408-4f0e-b13a-a629ed6a6421" containerName="revoke-keys" Oct 30 05:27:47.584355 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.584333 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:47.613463 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.604161 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9"] Oct 30 05:27:47.613693 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.612658 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:27:47.613811 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.613214 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:27:47.613906 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.613273 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:27:47.613996 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.613366 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:27:47.659940 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.659903 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6b76\" (UniqueName: \"kubernetes.io/projected/d0595324-4f00-471e-8acc-b498ce5dd5f6-kube-api-access-x6b76\") pod \"maas-api-revoke-sub-0b7e8edb-4nzb9\" (UID: \"d0595324-4f00-471e-8acc-b498ce5dd5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:47.660212 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.660190 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4nzb9\" (UID: \"d0595324-4f00-471e-8acc-b498ce5dd5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:47.660387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.660370 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4nzb9\" (UID: \"d0595324-4f00-471e-8acc-b498ce5dd5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:47.770048 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.770004 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6b76\" (UniqueName: \"kubernetes.io/projected/d0595324-4f00-471e-8acc-b498ce5dd5f6-kube-api-access-x6b76\") pod \"maas-api-revoke-sub-0b7e8edb-4nzb9\" (UID: \"d0595324-4f00-471e-8acc-b498ce5dd5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:47.770400 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.770375 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4nzb9\" (UID: \"d0595324-4f00-471e-8acc-b498ce5dd5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:47.770609 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.770587 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4nzb9\" (UID: \"d0595324-4f00-471e-8acc-b498ce5dd5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:47.771630 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.771605 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4nzb9\" (UID: \"d0595324-4f00-471e-8acc-b498ce5dd5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:47.779853 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.779827 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-4nzb9\" (UID: \"d0595324-4f00-471e-8acc-b498ce5dd5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:47.827369 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.827325 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6b76\" (UniqueName: \"kubernetes.io/projected/d0595324-4f00-471e-8acc-b498ce5dd5f6-kube-api-access-x6b76\") pod \"maas-api-revoke-sub-0b7e8edb-4nzb9\" (UID: \"d0595324-4f00-471e-8acc-b498ce5dd5f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:47.951313 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:47.950780 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:48.491510 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:48.490108 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9"] Oct 30 05:27:48.537299 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:27:48.537145 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0595324_4f00_471e_8acc_b498ce5dd5f6.slice/crio-7f42980f5dfec044b560fd1486c448b3ca68804f184b073a8e51ddc408b06cf1 WatchSource:0}: Error finding container 7f42980f5dfec044b560fd1486c448b3ca68804f184b073a8e51ddc408b06cf1: Status 404 returned error can't find the container with id 7f42980f5dfec044b560fd1486c448b3ca68804f184b073a8e51ddc408b06cf1 Oct 30 05:27:49.045640 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:49.045600 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" event={"ID":"d0595324-4f00-471e-8acc-b498ce5dd5f6","Type":"ContainerStarted","Data":"b46d3e8d9273a26fcc586032567d604598926300ac9b65c71ca95f90acb47dd0"} Oct 30 05:27:49.046225 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:49.046198 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" event={"ID":"d0595324-4f00-471e-8acc-b498ce5dd5f6","Type":"ContainerStarted","Data":"7f42980f5dfec044b560fd1486c448b3ca68804f184b073a8e51ddc408b06cf1"} Oct 30 05:27:49.105446 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:49.103823 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" podStartSLOduration=2.103792802 podStartE2EDuration="2.103792802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:27:49.10081877 +0000 UTC m=+5392.386693576" watchObservedRunningTime="2026-10-30 05:27:49.103792802 +0000 UTC m=+5392.389667577" Oct 30 05:27:50.052555 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:50.052512 4793 generic.go:356] "Generic (PLEG): container finished" podID="d0595324-4f00-471e-8acc-b498ce5dd5f6" containerID="b46d3e8d9273a26fcc586032567d604598926300ac9b65c71ca95f90acb47dd0" exitCode=0 Oct 30 05:27:50.052976 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:50.052590 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" event={"ID":"d0595324-4f00-471e-8acc-b498ce5dd5f6","Type":"ContainerDied","Data":"b46d3e8d9273a26fcc586032567d604598926300ac9b65c71ca95f90acb47dd0"} Oct 30 05:27:51.201328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:51.201295 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:51.267845 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:51.267807 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0595324-4f00-471e-8acc-b498ce5dd5f6-kube-api-access-x6b76\" (UniqueName: \"kubernetes.io/projected/d0595324-4f00-471e-8acc-b498ce5dd5f6-kube-api-access-x6b76\") pod \"d0595324-4f00-471e-8acc-b498ce5dd5f6\" (UID: \"d0595324-4f00-471e-8acc-b498ce5dd5f6\") " Oct 30 05:27:51.268006 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:51.267865 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-openshift-ca\") pod \"d0595324-4f00-471e-8acc-b498ce5dd5f6\" (UID: \"d0595324-4f00-471e-8acc-b498ce5dd5f6\") " Oct 30 05:27:51.268083 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:51.268012 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-xks-ca\") pod \"d0595324-4f00-471e-8acc-b498ce5dd5f6\" (UID: \"d0595324-4f00-471e-8acc-b498ce5dd5f6\") " Oct 30 05:27:51.268257 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:51.268224 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-openshift-ca" pod "d0595324-4f00-471e-8acc-b498ce5dd5f6" (UID: "d0595324-4f00-471e-8acc-b498ce5dd5f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:27:51.268395 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:51.268381 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:27:51.270014 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:51.269990 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0595324-4f00-471e-8acc-b498ce5dd5f6-kube-api-access-x6b76" pod "d0595324-4f00-471e-8acc-b498ce5dd5f6" (UID: "d0595324-4f00-471e-8acc-b498ce5dd5f6"). InnerVolumeSpecName "kube-api-access-x6b76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:27:51.270462 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:51.270433 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-xks-ca" pod "d0595324-4f00-471e-8acc-b498ce5dd5f6" (UID: "d0595324-4f00-471e-8acc-b498ce5dd5f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:27:51.369062 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:51.368986 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0595324-4f00-471e-8acc-b498ce5dd5f6-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:27:51.369062 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:51.369018 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6b76\" (UniqueName: \"kubernetes.io/projected/d0595324-4f00-471e-8acc-b498ce5dd5f6-kube-api-access-x6b76\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:27:52.065879 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:52.065852 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" Oct 30 05:27:52.066097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:52.065847 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9" event={"ID":"d0595324-4f00-471e-8acc-b498ce5dd5f6","Type":"ContainerDied","Data":"7f42980f5dfec044b560fd1486c448b3ca68804f184b073a8e51ddc408b06cf1"} Oct 30 05:27:52.066097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:52.065974 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f42980f5dfec044b560fd1486c448b3ca68804f184b073a8e51ddc408b06cf1" Oct 30 05:27:58.341470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.341435 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx"] Oct 30 05:27:58.341863 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.341815 4793 cpu_manager.go:414] "containerMap: removing container" podUID="d0595324-4f00-471e-8acc-b498ce5dd5f6" containerName="revoke-keys" Oct 30 05:27:58.341863 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.341827 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0595324-4f00-471e-8acc-b498ce5dd5f6" containerName="revoke-keys" Oct 30 05:27:58.342102 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.341904 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0595324-4f00-471e-8acc-b498ce5dd5f6" containerName="revoke-keys" Oct 30 05:27:58.345220 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.345198 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:27:58.385006 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.384968 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx"] Oct 30 05:27:58.389346 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.389318 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:27:58.389346 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.389323 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:27:58.389603 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.389331 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:27:58.389750 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.389730 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:27:58.435537 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.435507 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82625169-e962-440e-826b-592778f42525-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9sfdx\" (UID: \"82625169-e962-440e-826b-592778f42525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:27:58.435682 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.435644 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkxwf\" (UniqueName: \"kubernetes.io/projected/82625169-e962-440e-826b-592778f42525-kube-api-access-lkxwf\") pod \"maas-api-revoke-sub-3d4156f1-9sfdx\" (UID: \"82625169-e962-440e-826b-592778f42525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:27:58.435738 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.435703 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82625169-e962-440e-826b-592778f42525-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9sfdx\" (UID: \"82625169-e962-440e-826b-592778f42525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:27:58.536439 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.536384 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkxwf\" (UniqueName: \"kubernetes.io/projected/82625169-e962-440e-826b-592778f42525-kube-api-access-lkxwf\") pod \"maas-api-revoke-sub-3d4156f1-9sfdx\" (UID: \"82625169-e962-440e-826b-592778f42525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:27:58.536577 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.536462 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82625169-e962-440e-826b-592778f42525-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9sfdx\" (UID: \"82625169-e962-440e-826b-592778f42525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:27:58.536577 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.536537 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82625169-e962-440e-826b-592778f42525-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9sfdx\" (UID: \"82625169-e962-440e-826b-592778f42525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:27:58.537104 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.537084 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82625169-e962-440e-826b-592778f42525-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9sfdx\" (UID: \"82625169-e962-440e-826b-592778f42525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:27:58.538793 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.538763 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82625169-e962-440e-826b-592778f42525-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9sfdx\" (UID: \"82625169-e962-440e-826b-592778f42525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:27:58.559921 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.559896 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkxwf\" (UniqueName: \"kubernetes.io/projected/82625169-e962-440e-826b-592778f42525-kube-api-access-lkxwf\") pod \"maas-api-revoke-sub-3d4156f1-9sfdx\" (UID: \"82625169-e962-440e-826b-592778f42525\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:27:58.695366 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.695343 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:27:58.993080 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:58.993051 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx"] Oct 30 05:27:58.993886 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:27:58.993863 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82625169_e962_440e_826b_592778f42525.slice/crio-d096c6f3f9a570aabfd3a0cc7ff50fdcc750426805fbce5b6f53e4e6f3faad05 WatchSource:0}: Error finding container d096c6f3f9a570aabfd3a0cc7ff50fdcc750426805fbce5b6f53e4e6f3faad05: Status 404 returned error can't find the container with id d096c6f3f9a570aabfd3a0cc7ff50fdcc750426805fbce5b6f53e4e6f3faad05 Oct 30 05:27:59.106852 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:27:59.106822 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" event={"ID":"82625169-e962-440e-826b-592778f42525","Type":"ContainerStarted","Data":"d096c6f3f9a570aabfd3a0cc7ff50fdcc750426805fbce5b6f53e4e6f3faad05"} Oct 30 05:28:00.114294 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:00.114261 4793 generic.go:356] "Generic (PLEG): container finished" podID="82625169-e962-440e-826b-592778f42525" containerID="4573d77d386069e5b81e7ff1d4500b19212869e603d2c0c8fa71661400fcb147" exitCode=0 Oct 30 05:28:00.114709 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:00.114323 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" event={"ID":"82625169-e962-440e-826b-592778f42525","Type":"ContainerDied","Data":"4573d77d386069e5b81e7ff1d4500b19212869e603d2c0c8fa71661400fcb147"} Oct 30 05:28:01.250279 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:01.250254 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:28:01.257753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:01.257720 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:28:01.362542 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:01.362511 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82625169-e962-440e-826b-592778f42525-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82625169-e962-440e-826b-592778f42525-maas-api-openshift-ca\") pod \"82625169-e962-440e-826b-592778f42525\" (UID: \"82625169-e962-440e-826b-592778f42525\") " Oct 30 05:28:01.362707 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:01.362564 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82625169-e962-440e-826b-592778f42525-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82625169-e962-440e-826b-592778f42525-maas-api-xks-ca\") pod \"82625169-e962-440e-826b-592778f42525\" (UID: \"82625169-e962-440e-826b-592778f42525\") " Oct 30 05:28:01.362707 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:01.362604 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82625169-e962-440e-826b-592778f42525-kube-api-access-lkxwf\" (UniqueName: \"kubernetes.io/projected/82625169-e962-440e-826b-592778f42525-kube-api-access-lkxwf\") pod \"82625169-e962-440e-826b-592778f42525\" (UID: \"82625169-e962-440e-826b-592778f42525\") " Oct 30 05:28:01.362874 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:01.362850 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82625169-e962-440e-826b-592778f42525-maas-api-openshift-ca" pod "82625169-e962-440e-826b-592778f42525" (UID: "82625169-e962-440e-826b-592778f42525"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:28:01.364642 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:01.364618 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82625169-e962-440e-826b-592778f42525-kube-api-access-lkxwf" pod "82625169-e962-440e-826b-592778f42525" (UID: "82625169-e962-440e-826b-592778f42525"). InnerVolumeSpecName "kube-api-access-lkxwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:28:01.364781 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:01.364764 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82625169-e962-440e-826b-592778f42525-maas-api-xks-ca" pod "82625169-e962-440e-826b-592778f42525" (UID: "82625169-e962-440e-826b-592778f42525"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:28:01.463562 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:01.463535 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82625169-e962-440e-826b-592778f42525-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:28:01.463562 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:01.463559 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82625169-e962-440e-826b-592778f42525-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:28:01.463729 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:01.463570 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkxwf\" (UniqueName: \"kubernetes.io/projected/82625169-e962-440e-826b-592778f42525-kube-api-access-lkxwf\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:28:01.499853 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:01.499824 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:28:02.126111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:02.126084 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" Oct 30 05:28:02.126111 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:02.126096 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx" event={"ID":"82625169-e962-440e-826b-592778f42525","Type":"ContainerDied","Data":"d096c6f3f9a570aabfd3a0cc7ff50fdcc750426805fbce5b6f53e4e6f3faad05"} Oct 30 05:28:02.126335 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:02.126138 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d096c6f3f9a570aabfd3a0cc7ff50fdcc750426805fbce5b6f53e4e6f3faad05" Oct 30 05:28:05.117051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:05.117012 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:28:11.230128 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:28:11.230085 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:28:16.898009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:16.897979 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx"] Oct 30 05:28:16.909369 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:16.909344 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9sfdx"] Oct 30 05:28:16.950209 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:16.950170 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448"] Oct 30 05:28:16.950590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:16.950577 4793 cpu_manager.go:414] "containerMap: removing container" podUID="82625169-e962-440e-826b-592778f42525" containerName="revoke-keys" Oct 30 05:28:16.950643 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:16.950592 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82625169-e962-440e-826b-592778f42525" containerName="revoke-keys" Oct 30 05:28:16.950681 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:16.950658 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="82625169-e962-440e-826b-592778f42525" containerName="revoke-keys" Oct 30 05:28:16.953765 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:16.953745 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:16.970104 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:16.970077 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:28:16.970323 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:16.970299 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:28:16.975894 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:16.975866 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448"] Oct 30 05:28:16.981410 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:16.981390 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:28:16.981678 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:16.981654 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:28:17.071716 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:17.071683 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fl448\" (UID: \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:17.071865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:17.071799 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6t59\" (UniqueName: \"kubernetes.io/projected/00bc668b-f27f-43be-b8c2-1e228dab5fbd-kube-api-access-v6t59\") pod \"maas-api-revoke-sub-3d4156f1-fl448\" (UID: \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:17.071865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:17.071831 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fl448\" (UID: \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:17.173403 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:17.173329 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6t59\" (UniqueName: \"kubernetes.io/projected/00bc668b-f27f-43be-b8c2-1e228dab5fbd-kube-api-access-v6t59\") pod \"maas-api-revoke-sub-3d4156f1-fl448\" (UID: \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:17.173403 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:17.173382 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fl448\" (UID: \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:17.173633 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:17.173453 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fl448\" (UID: \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:17.174076 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:17.174047 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fl448\" (UID: \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:17.175813 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:17.175793 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fl448\" (UID: \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:17.196211 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:17.196184 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6t59\" (UniqueName: \"kubernetes.io/projected/00bc668b-f27f-43be-b8c2-1e228dab5fbd-kube-api-access-v6t59\") pod \"maas-api-revoke-sub-3d4156f1-fl448\" (UID: \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:17.270027 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:17.269988 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:17.569331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:17.569309 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448"] Oct 30 05:28:17.571461 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:28:17.571408 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bc668b_f27f_43be_b8c2_1e228dab5fbd.slice/crio-bc1d73812eba2db3471e0579acec3d9746eee4cf2b174e5cf1e29993914c501e WatchSource:0}: Error finding container bc1d73812eba2db3471e0579acec3d9746eee4cf2b174e5cf1e29993914c501e: Status 404 returned error can't find the container with id bc1d73812eba2db3471e0579acec3d9746eee4cf2b174e5cf1e29993914c501e Oct 30 05:28:18.257222 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:18.257178 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" event={"ID":"00bc668b-f27f-43be-b8c2-1e228dab5fbd","Type":"ContainerStarted","Data":"eb65864ec19a14138bc7bfcfbab111de05159af5f005da403243dd8850948766"} Oct 30 05:28:18.257651 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:18.257245 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" event={"ID":"00bc668b-f27f-43be-b8c2-1e228dab5fbd","Type":"ContainerStarted","Data":"bc1d73812eba2db3471e0579acec3d9746eee4cf2b174e5cf1e29993914c501e"} Oct 30 05:28:18.298575 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:18.298526 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" podStartSLOduration=2.298512339 podStartE2EDuration="2.298512339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:28:18.29668442 +0000 UTC m=+5421.582559195" watchObservedRunningTime="2026-10-30 05:28:18.298512339 +0000 UTC m=+5421.584387114" Oct 30 05:28:18.844702 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:18.844670 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82625169-e962-440e-826b-592778f42525" path="/var/lib/kubelet/pods/82625169-e962-440e-826b-592778f42525/volumes" Oct 30 05:28:19.265098 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:19.265058 4793 generic.go:356] "Generic (PLEG): container finished" podID="00bc668b-f27f-43be-b8c2-1e228dab5fbd" containerID="eb65864ec19a14138bc7bfcfbab111de05159af5f005da403243dd8850948766" exitCode=0 Oct 30 05:28:19.265529 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:19.265148 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" event={"ID":"00bc668b-f27f-43be-b8c2-1e228dab5fbd","Type":"ContainerDied","Data":"eb65864ec19a14138bc7bfcfbab111de05159af5f005da403243dd8850948766"} Oct 30 05:28:20.410782 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:20.410757 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:20.503748 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:20.503719 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00bc668b-f27f-43be-b8c2-1e228dab5fbd-kube-api-access-v6t59\" (UniqueName: \"kubernetes.io/projected/00bc668b-f27f-43be-b8c2-1e228dab5fbd-kube-api-access-v6t59\") pod \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\" (UID: \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\") " Oct 30 05:28:20.503906 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:20.503768 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-openshift-ca\") pod \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\" (UID: \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\") " Oct 30 05:28:20.503906 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:20.503819 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-xks-ca\") pod \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\" (UID: \"00bc668b-f27f-43be-b8c2-1e228dab5fbd\") " Oct 30 05:28:20.504137 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:20.504118 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-openshift-ca" pod "00bc668b-f27f-43be-b8c2-1e228dab5fbd" (UID: "00bc668b-f27f-43be-b8c2-1e228dab5fbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:28:20.505935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:20.505907 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-xks-ca" pod "00bc668b-f27f-43be-b8c2-1e228dab5fbd" (UID: "00bc668b-f27f-43be-b8c2-1e228dab5fbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:28:20.506052 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:20.505999 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00bc668b-f27f-43be-b8c2-1e228dab5fbd-kube-api-access-v6t59" pod "00bc668b-f27f-43be-b8c2-1e228dab5fbd" (UID: "00bc668b-f27f-43be-b8c2-1e228dab5fbd"). InnerVolumeSpecName "kube-api-access-v6t59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:28:20.604995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:20.604918 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:28:20.604995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:20.604945 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6t59\" (UniqueName: \"kubernetes.io/projected/00bc668b-f27f-43be-b8c2-1e228dab5fbd-kube-api-access-v6t59\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:28:20.604995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:20.604954 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00bc668b-f27f-43be-b8c2-1e228dab5fbd-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:28:21.278753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:21.278718 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" Oct 30 05:28:21.278943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:21.278721 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448" event={"ID":"00bc668b-f27f-43be-b8c2-1e228dab5fbd","Type":"ContainerDied","Data":"bc1d73812eba2db3471e0579acec3d9746eee4cf2b174e5cf1e29993914c501e"} Oct 30 05:28:21.278943 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:21.278869 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc1d73812eba2db3471e0579acec3d9746eee4cf2b174e5cf1e29993914c501e" Oct 30 05:28:41.218925 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:28:41.218839 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:28:47.672450 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.672395 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4"] Oct 30 05:28:47.673062 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.672898 4793 cpu_manager.go:414] "containerMap: removing container" podUID="00bc668b-f27f-43be-b8c2-1e228dab5fbd" containerName="revoke-keys" Oct 30 05:28:47.673062 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.672915 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00bc668b-f27f-43be-b8c2-1e228dab5fbd" containerName="revoke-keys" Oct 30 05:28:47.673062 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.672992 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="00bc668b-f27f-43be-b8c2-1e228dab5fbd" containerName="revoke-keys" Oct 30 05:28:47.676119 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.676094 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:47.703744 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.703693 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:28:47.703936 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.703726 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:28:47.704032 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.703722 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:28:47.704032 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.703746 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:28:47.707350 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.707323 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4"] Oct 30 05:28:47.780638 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.780607 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr8gn\" (UniqueName: \"kubernetes.io/projected/a820726d-168a-449c-9c87-6e290978a4cb-kube-api-access-gr8gn\") pod \"maas-api-revoke-sub-e2fc9c5b-szwq4\" (UID: \"a820726d-168a-449c-9c87-6e290978a4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:47.780816 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.780758 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-szwq4\" (UID: \"a820726d-168a-449c-9c87-6e290978a4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:47.780865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.780826 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-szwq4\" (UID: \"a820726d-168a-449c-9c87-6e290978a4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:47.881498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.881457 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-szwq4\" (UID: \"a820726d-168a-449c-9c87-6e290978a4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:47.881669 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.881511 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-szwq4\" (UID: \"a820726d-168a-449c-9c87-6e290978a4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:47.881669 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.881553 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8gn\" (UniqueName: \"kubernetes.io/projected/a820726d-168a-449c-9c87-6e290978a4cb-kube-api-access-gr8gn\") pod \"maas-api-revoke-sub-e2fc9c5b-szwq4\" (UID: \"a820726d-168a-449c-9c87-6e290978a4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:47.882215 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.882193 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-szwq4\" (UID: \"a820726d-168a-449c-9c87-6e290978a4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:47.883794 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.883772 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-szwq4\" (UID: \"a820726d-168a-449c-9c87-6e290978a4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:47.904832 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.904810 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8gn\" (UniqueName: \"kubernetes.io/projected/a820726d-168a-449c-9c87-6e290978a4cb-kube-api-access-gr8gn\") pod \"maas-api-revoke-sub-e2fc9c5b-szwq4\" (UID: \"a820726d-168a-449c-9c87-6e290978a4cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:47.992302 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:47.992280 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:48.289779 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:48.289750 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4"] Oct 30 05:28:48.291120 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:28:48.291092 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda820726d_168a_449c_9c87_6e290978a4cb.slice/crio-df332a7b0e34af5da4844fddba64552450649018e1a4d9b53c6bd6f3c4d9bbba WatchSource:0}: Error finding container df332a7b0e34af5da4844fddba64552450649018e1a4d9b53c6bd6f3c4d9bbba: Status 404 returned error can't find the container with id df332a7b0e34af5da4844fddba64552450649018e1a4d9b53c6bd6f3c4d9bbba Oct 30 05:28:48.449359 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:48.449329 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" event={"ID":"a820726d-168a-449c-9c87-6e290978a4cb","Type":"ContainerStarted","Data":"df332a7b0e34af5da4844fddba64552450649018e1a4d9b53c6bd6f3c4d9bbba"} Oct 30 05:28:49.456052 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:49.456020 4793 generic.go:356] "Generic (PLEG): container finished" podID="a820726d-168a-449c-9c87-6e290978a4cb" containerID="1e739a8535a2a7b3f1406ad3f86427a42de5e7f7aafeaca76f9a3a5fb1ad7983" exitCode=0 Oct 30 05:28:49.456465 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:49.456111 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" event={"ID":"a820726d-168a-449c-9c87-6e290978a4cb","Type":"ContainerDied","Data":"1e739a8535a2a7b3f1406ad3f86427a42de5e7f7aafeaca76f9a3a5fb1ad7983"} Oct 30 05:28:50.591319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:50.591296 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:50.608233 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:50.608209 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-openshift-ca\") pod \"a820726d-168a-449c-9c87-6e290978a4cb\" (UID: \"a820726d-168a-449c-9c87-6e290978a4cb\") " Oct 30 05:28:50.608345 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:50.608267 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a820726d-168a-449c-9c87-6e290978a4cb-kube-api-access-gr8gn\" (UniqueName: \"kubernetes.io/projected/a820726d-168a-449c-9c87-6e290978a4cb-kube-api-access-gr8gn\") pod \"a820726d-168a-449c-9c87-6e290978a4cb\" (UID: \"a820726d-168a-449c-9c87-6e290978a4cb\") " Oct 30 05:28:50.608345 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:50.608310 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-xks-ca\") pod \"a820726d-168a-449c-9c87-6e290978a4cb\" (UID: \"a820726d-168a-449c-9c87-6e290978a4cb\") " Oct 30 05:28:50.608573 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:50.608552 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-openshift-ca" pod "a820726d-168a-449c-9c87-6e290978a4cb" (UID: "a820726d-168a-449c-9c87-6e290978a4cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:28:50.610266 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:50.610246 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a820726d-168a-449c-9c87-6e290978a4cb-kube-api-access-gr8gn" pod "a820726d-168a-449c-9c87-6e290978a4cb" (UID: "a820726d-168a-449c-9c87-6e290978a4cb"). InnerVolumeSpecName "kube-api-access-gr8gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:28:50.610377 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:50.610356 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-xks-ca" pod "a820726d-168a-449c-9c87-6e290978a4cb" (UID: "a820726d-168a-449c-9c87-6e290978a4cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:28:50.709582 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:50.709554 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:28:50.709582 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:50.709577 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr8gn\" (UniqueName: \"kubernetes.io/projected/a820726d-168a-449c-9c87-6e290978a4cb-kube-api-access-gr8gn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:28:50.709582 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:50.709587 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a820726d-168a-449c-9c87-6e290978a4cb-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:28:51.469530 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:51.469494 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" event={"ID":"a820726d-168a-449c-9c87-6e290978a4cb","Type":"ContainerDied","Data":"df332a7b0e34af5da4844fddba64552450649018e1a4d9b53c6bd6f3c4d9bbba"} Oct 30 05:28:51.469530 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:51.469526 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4" Oct 30 05:28:51.469748 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:51.469542 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df332a7b0e34af5da4844fddba64552450649018e1a4d9b53c6bd6f3c4d9bbba" Oct 30 05:28:58.047447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.047402 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8"] Oct 30 05:28:58.047869 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.047807 4793 cpu_manager.go:414] "containerMap: removing container" podUID="a820726d-168a-449c-9c87-6e290978a4cb" containerName="revoke-keys" Oct 30 05:28:58.047869 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.047820 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a820726d-168a-449c-9c87-6e290978a4cb" containerName="revoke-keys" Oct 30 05:28:58.047968 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.047905 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="a820726d-168a-449c-9c87-6e290978a4cb" containerName="revoke-keys" Oct 30 05:28:58.054951 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.054921 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:28:58.070030 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.070004 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8"] Oct 30 05:28:58.078219 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.078191 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:28:58.078485 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.078460 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:28:58.125018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.124993 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:28:58.125018 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.124998 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:28:58.175689 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.175664 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5tk\" (UniqueName: \"kubernetes.io/projected/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-kube-api-access-wr5tk\") pod \"maas-api-revoke-sub-5a29efb3-g8qw8\" (UID: \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:28:58.175845 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.175710 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-g8qw8\" (UID: \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:28:58.175896 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.175838 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-g8qw8\" (UID: \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:28:58.278429 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.278325 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-g8qw8\" (UID: \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:28:58.278780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.278755 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-g8qw8\" (UID: \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:28:58.278951 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.278935 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5tk\" (UniqueName: \"kubernetes.io/projected/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-kube-api-access-wr5tk\") pod \"maas-api-revoke-sub-5a29efb3-g8qw8\" (UID: \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:28:58.280393 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.280365 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-g8qw8\" (UID: \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:28:58.284078 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.283460 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-g8qw8\" (UID: \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:28:58.302691 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.302638 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5tk\" (UniqueName: \"kubernetes.io/projected/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-kube-api-access-wr5tk\") pod \"maas-api-revoke-sub-5a29efb3-g8qw8\" (UID: \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:28:58.368234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.368209 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:28:58.669956 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:58.669924 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8"] Oct 30 05:28:58.671071 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:28:58.671042 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae4cd8dc_bcc7_4a79_9e5e_1c98df955ca9.slice/crio-050d2f80255d2910292b7f3aaf97f7fb8bede0134758608b273cafdd2bc78156 WatchSource:0}: Error finding container 050d2f80255d2910292b7f3aaf97f7fb8bede0134758608b273cafdd2bc78156: Status 404 returned error can't find the container with id 050d2f80255d2910292b7f3aaf97f7fb8bede0134758608b273cafdd2bc78156 Oct 30 05:28:59.518132 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:59.518099 4793 generic.go:356] "Generic (PLEG): container finished" podID="ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9" containerID="6844647d82d195c389ef43bfcda0022b0a861ed3c1742b337e04facc04eb3fdd" exitCode=0 Oct 30 05:28:59.518539 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:59.518146 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" event={"ID":"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9","Type":"ContainerDied","Data":"6844647d82d195c389ef43bfcda0022b0a861ed3c1742b337e04facc04eb3fdd"} Oct 30 05:28:59.518539 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:28:59.518198 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" event={"ID":"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9","Type":"ContainerStarted","Data":"050d2f80255d2910292b7f3aaf97f7fb8bede0134758608b273cafdd2bc78156"} Oct 30 05:29:00.923447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:00.922698 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:29:01.027755 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.027715 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-kube-api-access-wr5tk\" (UniqueName: \"kubernetes.io/projected/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-kube-api-access-wr5tk\") pod \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\" (UID: \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\") " Oct 30 05:29:01.032105 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.028038 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-openshift-ca\") pod \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\" (UID: \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\") " Oct 30 05:29:01.032105 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.031284 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-xks-ca\") pod \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\" (UID: \"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9\") " Oct 30 05:29:01.032105 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.031140 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-openshift-ca" pod "ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9" (UID: "ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:29:01.032105 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.032064 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:29:01.033205 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.033179 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-kube-api-access-wr5tk" pod "ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9" (UID: "ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9"). InnerVolumeSpecName "kube-api-access-wr5tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:29:01.035382 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.035349 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-xks-ca" pod "ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9" (UID: "ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:29:01.134097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.133982 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr5tk\" (UniqueName: \"kubernetes.io/projected/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-kube-api-access-wr5tk\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:29:01.134097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.134018 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:29:01.404319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.404225 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:01.539898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.539545 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" event={"ID":"ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9","Type":"ContainerDied","Data":"050d2f80255d2910292b7f3aaf97f7fb8bede0134758608b273cafdd2bc78156"} Oct 30 05:29:01.539898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.539610 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="050d2f80255d2910292b7f3aaf97f7fb8bede0134758608b273cafdd2bc78156" Oct 30 05:29:01.539898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:01.539708 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8" Oct 30 05:29:11.219578 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:29:11.219536 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:29:13.966110 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:13.966068 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448"] Oct 30 05:29:13.976832 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:13.976806 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fl448"] Oct 30 05:29:14.019315 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.019279 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp"] Oct 30 05:29:14.019722 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.019705 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9" containerName="revoke-keys" Oct 30 05:29:14.019722 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.019722 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9" containerName="revoke-keys" Oct 30 05:29:14.019883 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.019815 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9" containerName="revoke-keys" Oct 30 05:29:14.023280 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.023256 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:14.038918 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.038226 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:29:14.038918 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.038311 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:29:14.038918 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.038352 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:29:14.038918 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.038404 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:29:14.040502 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.040482 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp"] Oct 30 05:29:14.149807 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.149777 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmll9\" (UniqueName: \"kubernetes.io/projected/fbbec7af-1642-4f51-b128-187b2a078846-kube-api-access-cmll9\") pod \"maas-api-revoke-sub-3d4156f1-88bpp\" (UID: \"fbbec7af-1642-4f51-b128-187b2a078846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:14.149983 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.149829 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-88bpp\" (UID: \"fbbec7af-1642-4f51-b128-187b2a078846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:14.149983 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.149945 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-88bpp\" (UID: \"fbbec7af-1642-4f51-b128-187b2a078846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:14.250793 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.250725 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-88bpp\" (UID: \"fbbec7af-1642-4f51-b128-187b2a078846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:14.250793 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.250770 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmll9\" (UniqueName: \"kubernetes.io/projected/fbbec7af-1642-4f51-b128-187b2a078846-kube-api-access-cmll9\") pod \"maas-api-revoke-sub-3d4156f1-88bpp\" (UID: \"fbbec7af-1642-4f51-b128-187b2a078846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:14.251026 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.250812 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-88bpp\" (UID: \"fbbec7af-1642-4f51-b128-187b2a078846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:14.251454 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.251408 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-88bpp\" (UID: \"fbbec7af-1642-4f51-b128-187b2a078846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:14.253058 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.253035 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-88bpp\" (UID: \"fbbec7af-1642-4f51-b128-187b2a078846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:14.273311 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.273288 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmll9\" (UniqueName: \"kubernetes.io/projected/fbbec7af-1642-4f51-b128-187b2a078846-kube-api-access-cmll9\") pod \"maas-api-revoke-sub-3d4156f1-88bpp\" (UID: \"fbbec7af-1642-4f51-b128-187b2a078846\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:14.337498 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.337474 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:14.632901 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.632878 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp"] Oct 30 05:29:14.634023 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:29:14.633994 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbbec7af_1642_4f51_b128_187b2a078846.slice/crio-bc7f8a3d56a411318cac4539b52276c3220f8f3f19aae7d01ce863d3840d2141 WatchSource:0}: Error finding container bc7f8a3d56a411318cac4539b52276c3220f8f3f19aae7d01ce863d3840d2141: Status 404 returned error can't find the container with id bc7f8a3d56a411318cac4539b52276c3220f8f3f19aae7d01ce863d3840d2141 Oct 30 05:29:14.843640 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:14.843591 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00bc668b-f27f-43be-b8c2-1e228dab5fbd" path="/var/lib/kubelet/pods/00bc668b-f27f-43be-b8c2-1e228dab5fbd/volumes" Oct 30 05:29:15.625171 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:15.625136 4793 generic.go:356] "Generic (PLEG): container finished" podID="fbbec7af-1642-4f51-b128-187b2a078846" containerID="96e814daa50018829090bb73d26811c2ad417bcb04ce6efb533d359cc0c84101" exitCode=0 Oct 30 05:29:15.625602 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:15.625220 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" event={"ID":"fbbec7af-1642-4f51-b128-187b2a078846","Type":"ContainerDied","Data":"96e814daa50018829090bb73d26811c2ad417bcb04ce6efb533d359cc0c84101"} Oct 30 05:29:15.625602 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:15.625270 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" event={"ID":"fbbec7af-1642-4f51-b128-187b2a078846","Type":"ContainerStarted","Data":"bc7f8a3d56a411318cac4539b52276c3220f8f3f19aae7d01ce863d3840d2141"} Oct 30 05:29:16.759796 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:16.759769 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:16.874588 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:16.874558 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-xks-ca\") pod \"fbbec7af-1642-4f51-b128-187b2a078846\" (UID: \"fbbec7af-1642-4f51-b128-187b2a078846\") " Oct 30 05:29:16.874768 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:16.874603 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbbec7af-1642-4f51-b128-187b2a078846-kube-api-access-cmll9\" (UniqueName: \"kubernetes.io/projected/fbbec7af-1642-4f51-b128-187b2a078846-kube-api-access-cmll9\") pod \"fbbec7af-1642-4f51-b128-187b2a078846\" (UID: \"fbbec7af-1642-4f51-b128-187b2a078846\") " Oct 30 05:29:16.874768 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:16.874654 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-openshift-ca\") pod \"fbbec7af-1642-4f51-b128-187b2a078846\" (UID: \"fbbec7af-1642-4f51-b128-187b2a078846\") " Oct 30 05:29:16.875080 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:16.875056 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-openshift-ca" pod "fbbec7af-1642-4f51-b128-187b2a078846" (UID: "fbbec7af-1642-4f51-b128-187b2a078846"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:29:16.876557 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:16.876529 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-xks-ca" pod "fbbec7af-1642-4f51-b128-187b2a078846" (UID: "fbbec7af-1642-4f51-b128-187b2a078846"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:29:16.876692 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:16.876667 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbbec7af-1642-4f51-b128-187b2a078846-kube-api-access-cmll9" pod "fbbec7af-1642-4f51-b128-187b2a078846" (UID: "fbbec7af-1642-4f51-b128-187b2a078846"). InnerVolumeSpecName "kube-api-access-cmll9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:29:16.975701 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:16.975677 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:29:16.975701 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:16.975700 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmll9\" (UniqueName: \"kubernetes.io/projected/fbbec7af-1642-4f51-b128-187b2a078846-kube-api-access-cmll9\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:29:16.975859 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:16.975710 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbbec7af-1642-4f51-b128-187b2a078846-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:29:17.638092 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:17.638059 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" Oct 30 05:29:17.638274 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:17.638085 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp" event={"ID":"fbbec7af-1642-4f51-b128-187b2a078846","Type":"ContainerDied","Data":"bc7f8a3d56a411318cac4539b52276c3220f8f3f19aae7d01ce863d3840d2141"} Oct 30 05:29:17.638274 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:17.638137 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc7f8a3d56a411318cac4539b52276c3220f8f3f19aae7d01ce863d3840d2141" Oct 30 05:29:29.537235 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:29.537196 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:29.859369 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:29.859243 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn"] Oct 30 05:29:29.859984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:29.859958 4793 cpu_manager.go:414] "containerMap: removing container" podUID="fbbec7af-1642-4f51-b128-187b2a078846" containerName="revoke-keys" Oct 30 05:29:29.859984 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:29.859983 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbbec7af-1642-4f51-b128-187b2a078846" containerName="revoke-keys" Oct 30 05:29:29.860319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:29.860118 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbbec7af-1642-4f51-b128-187b2a078846" containerName="revoke-keys" Oct 30 05:29:29.863626 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:29.863603 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:29.882778 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:29.882747 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-042f4c-openshift-default-dockercfg-xgxtq\"" Oct 30 05:29:29.897456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:29.897430 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn"] Oct 30 05:29:30.023642 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.023605 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43fce029-53ab-4b52-b743-13803d84ffef-istiod-ca-cert\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.023896 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.023869 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-credential-socket\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.023995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.023910 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43fce029-53ab-4b52-b743-13803d84ffef-istio-podinfo\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.023995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.023949 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlc7j\" (UniqueName: \"kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-kube-api-access-xlc7j\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.024108 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.023996 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-envoy\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.024108 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.024060 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-socket\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.024220 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.024122 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-istio-token\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.024220 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.024152 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-data\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.024220 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.024212 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-certs\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.046167 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.046124 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:30.125351 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.125263 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-socket\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.125351 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.125312 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-istio-token\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.125596 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.125455 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-data\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.125596 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.125531 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-certs\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.125674 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.125602 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43fce029-53ab-4b52-b743-13803d84ffef-istiod-ca-cert\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.125674 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.125637 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-credential-socket\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.125774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.125674 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43fce029-53ab-4b52-b743-13803d84ffef-istio-podinfo\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.125774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.125717 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlc7j\" (UniqueName: \"kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-kube-api-access-xlc7j\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.125774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.125755 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-socket\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.125774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.125767 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-envoy\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.125956 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.125811 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-data\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.125956 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.125912 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-certs\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.126405 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.126379 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43fce029-53ab-4b52-b743-13803d84ffef-istiod-ca-cert\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.126405 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.126392 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-credential-socket\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.128106 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.128086 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-envoy\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.128287 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.128263 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43fce029-53ab-4b52-b743-13803d84ffef-istio-podinfo\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.148051 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.148021 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-istio-token\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.148451 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.148427 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlc7j\" (UniqueName: \"kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-kube-api-access-xlc7j\") pod \"e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.178623 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.178603 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:30.271646 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.271617 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:30.492316 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.492289 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn"] Oct 30 05:29:30.493988 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:29:30.493958 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fce029_53ab_4b52_b743_13803d84ffef.slice/crio-18efe97c09f3b6c9fe7f68d38b96a2cb027879fea16dceb67a3b5b6e671e6e84 WatchSource:0}: Error finding container 18efe97c09f3b6c9fe7f68d38b96a2cb027879fea16dceb67a3b5b6e671e6e84: Status 404 returned error can't find the container with id 18efe97c09f3b6c9fe7f68d38b96a2cb027879fea16dceb67a3b5b6e671e6e84 Oct 30 05:29:30.496412 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.496376 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:29:30.496580 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.496491 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:29:30.496580 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.496570 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:29:30.744961 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.744863 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" event={"ID":"43fce029-53ab-4b52-b743-13803d84ffef","Type":"ContainerStarted","Data":"7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164"} Oct 30 05:29:30.744961 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.744928 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" event={"ID":"43fce029-53ab-4b52-b743-13803d84ffef","Type":"ContainerStarted","Data":"18efe97c09f3b6c9fe7f68d38b96a2cb027879fea16dceb67a3b5b6e671e6e84"} Oct 30 05:29:30.792283 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:30.792230 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" podStartSLOduration=1.7922157520000002 podStartE2EDuration="1.792215752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:30.788773252 +0000 UTC m=+5494.074648027" watchObservedRunningTime="2026-10-30 05:29:30.792215752 +0000 UTC m=+5494.078090527" Oct 30 05:29:31.179639 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:31.179607 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:31.185126 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:31.185100 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:31.750855 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:31.750826 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:31.751671 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:31.751654 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:33.421690 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:33.421642 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:33.646441 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:33.646393 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:35.089176 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.089139 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:35.102511 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.102466 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82"] Oct 30 05:29:35.106142 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.106119 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" Oct 30 05:29:35.123508 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.123487 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-042f4c\"" Oct 30 05:29:35.133395 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.132402 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-d4169098-dockercfg-kjnvt\"" Oct 30 05:29:35.135491 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.135468 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82"] Oct 30 05:29:35.280555 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.280524 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986e35e1-95ca-44a3-8218-ffa8ae132fb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82\" (UID: \"986e35e1-95ca-44a3-8218-ffa8ae132fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" Oct 30 05:29:35.280723 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.280574 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brm8n\" (UniqueName: \"kubernetes.io/projected/986e35e1-95ca-44a3-8218-ffa8ae132fb3-kube-api-access-brm8n\") pod \"payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82\" (UID: \"986e35e1-95ca-44a3-8218-ffa8ae132fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" Oct 30 05:29:35.320690 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.320653 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:35.381809 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.381739 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986e35e1-95ca-44a3-8218-ffa8ae132fb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82\" (UID: \"986e35e1-95ca-44a3-8218-ffa8ae132fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" Oct 30 05:29:35.381809 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.381787 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brm8n\" (UniqueName: \"kubernetes.io/projected/986e35e1-95ca-44a3-8218-ffa8ae132fb3-kube-api-access-brm8n\") pod \"payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82\" (UID: \"986e35e1-95ca-44a3-8218-ffa8ae132fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" Oct 30 05:29:35.382467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.382412 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986e35e1-95ca-44a3-8218-ffa8ae132fb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82\" (UID: \"986e35e1-95ca-44a3-8218-ffa8ae132fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" Oct 30 05:29:35.404303 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.404278 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brm8n\" (UniqueName: \"kubernetes.io/projected/986e35e1-95ca-44a3-8218-ffa8ae132fb3-kube-api-access-brm8n\") pod \"payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82\" (UID: \"986e35e1-95ca-44a3-8218-ffa8ae132fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" Oct 30 05:29:35.420114 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.420092 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" Oct 30 05:29:35.719728 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.719694 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82"] Oct 30 05:29:35.720705 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:29:35.720676 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986e35e1_95ca_44a3_8218_ffa8ae132fb3.slice/crio-84727b81747a65cf1c0f99bf4b3a0a640750260dc27a76ec56c74e6d7463c5fa WatchSource:0}: Error finding container 84727b81747a65cf1c0f99bf4b3a0a640750260dc27a76ec56c74e6d7463c5fa: Status 404 returned error can't find the container with id 84727b81747a65cf1c0f99bf4b3a0a640750260dc27a76ec56c74e6d7463c5fa Oct 30 05:29:35.774770 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:35.774739 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" event={"ID":"986e35e1-95ca-44a3-8218-ffa8ae132fb3","Type":"ContainerStarted","Data":"84727b81747a65cf1c0f99bf4b3a0a640750260dc27a76ec56c74e6d7463c5fa"} Oct 30 05:29:36.786501 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:36.786466 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" event={"ID":"986e35e1-95ca-44a3-8218-ffa8ae132fb3","Type":"ContainerStarted","Data":"e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616"} Oct 30 05:29:36.786992 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:36.786561 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" Oct 30 05:29:36.830047 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:36.829990 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" podStartSLOduration=1.829977527 podStartE2EDuration="1.829977527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:36.826883081 +0000 UTC m=+5500.112757855" watchObservedRunningTime="2026-10-30 05:29:36.829977527 +0000 UTC m=+5500.115852293" Oct 30 05:29:36.865470 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:36.865439 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:38.729058 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:38.729018 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:40.539721 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:40.539684 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:41.218976 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:29:41.218926 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:29:42.350097 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:42.350061 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:44.159865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:44.159813 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:46.191284 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:46.191248 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:47.033368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:47.033268 4793 patch_prober.go:29] interesting pod/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": read tcp 10.133.0.2:58250->10.133.1.38:15021: read: connection reset by peer" start-of-body= Oct 30 05:29:47.033368 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:47.033345 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": read tcp 10.133.0.2:58250->10.133.1.38:15021: read: connection reset by peer" Oct 30 05:29:47.795385 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:47.795356 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" Oct 30 05:29:47.862956 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:47.862916 4793 generic.go:356] "Generic (PLEG): container finished" podID="43fce029-53ab-4b52-b743-13803d84ffef" containerID="7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164" exitCode=0 Oct 30 05:29:47.863113 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:47.862954 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" event={"ID":"43fce029-53ab-4b52-b743-13803d84ffef","Type":"ContainerDied","Data":"7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164"} Oct 30 05:29:47.863335 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:47.863316 4793 scope.go:122] "RemoveContainer" containerID="7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164" Oct 30 05:29:47.864688 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:47.864643 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:29:47.864778 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:47.864728 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:29:47.864778 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:47.864771 4793 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 05:29:47.933752 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:47.933725 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:48.162033 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:48.161937 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:48.742152 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:48.742114 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:48.880857 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:48.880820 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" event={"ID":"43fce029-53ab-4b52-b743-13803d84ffef","Type":"ContainerStarted","Data":"6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411"} Oct 30 05:29:48.977446 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:48.977389 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:49.179735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:49.179694 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:49.180093 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:49.180069 4793 patch_prober.go:29] interesting pod/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 05:29:49.180233 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:49.180136 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 05:29:49.797187 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:49.797153 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:50.179937 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:50.179900 4793 patch_prober.go:29] interesting pod/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 05:29:50.180326 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:50.179975 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 05:29:51.004180 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.004142 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8"] Oct 30 05:29:51.011605 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.011576 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.026607 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.026585 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8"] Oct 30 05:29:51.034392 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.034372 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-042f4c\"/\"simulator-single-main-042f4c-kserve-self-signed-certs\"" Oct 30 05:29:51.034617 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.034601 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-042f4c\"/\"openshift-service-ca.crt\"" Oct 30 05:29:51.043268 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.043252 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-042f4c\"/\"default-dockercfg-vgzxh\"" Oct 30 05:29:51.043477 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.043459 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-042f4c\"/\"kube-root-ca.crt\"" Oct 30 05:29:51.179844 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.179810 4793 patch_prober.go:29] interesting pod/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 05:29:51.180053 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.179872 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 05:29:51.185360 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.185336 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-home\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.185503 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.185403 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-tmp-dir\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.185564 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.185495 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-dshm\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.185564 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.185526 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-model-cache\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.185680 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.185562 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8zpn\" (UniqueName: \"kubernetes.io/projected/e999f18e-33dd-4050-b3ed-48ae27eed876-kube-api-access-l8zpn\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.185680 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.185610 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e999f18e-33dd-4050-b3ed-48ae27eed876-tls-certs\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.200721 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.200687 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:51.292497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.291941 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-home\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.292497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.292055 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-tmp-dir\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.292497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.292092 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-dshm\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.292497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.292124 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-model-cache\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.292497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.292168 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8zpn\" (UniqueName: \"kubernetes.io/projected/e999f18e-33dd-4050-b3ed-48ae27eed876-kube-api-access-l8zpn\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.292497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.292233 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e999f18e-33dd-4050-b3ed-48ae27eed876-tls-certs\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.295514 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.293356 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-home\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.295514 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.293629 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-model-cache\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.295514 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.294186 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-tmp-dir\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.298285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.297964 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e999f18e-33dd-4050-b3ed-48ae27eed876-tls-certs\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.298285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.298193 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-dshm\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.316966 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.316942 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8zpn\" (UniqueName: \"kubernetes.io/projected/e999f18e-33dd-4050-b3ed-48ae27eed876-kube-api-access-l8zpn\") pod \"simulator-single-main-042f4c-kserve-57ff664d95-fwsb8\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.322737 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.322716 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:51.648357 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.648328 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8"] Oct 30 05:29:51.649680 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:29:51.649650 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode999f18e_33dd_4050_b3ed_48ae27eed876.slice/crio-86832abec47641cfe41908140ebae9d7bc7d5b11fd1a3da445527907d13e09d2 WatchSource:0}: Error finding container 86832abec47641cfe41908140ebae9d7bc7d5b11fd1a3da445527907d13e09d2: Status 404 returned error can't find the container with id 86832abec47641cfe41908140ebae9d7bc7d5b11fd1a3da445527907d13e09d2 Oct 30 05:29:51.676673 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.676640 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:51.900101 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.900012 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" event={"ID":"e999f18e-33dd-4050-b3ed-48ae27eed876","Type":"ContainerStarted","Data":"4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6"} Oct 30 05:29:51.900101 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.900058 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" event={"ID":"e999f18e-33dd-4050-b3ed-48ae27eed876","Type":"ContainerStarted","Data":"86832abec47641cfe41908140ebae9d7bc7d5b11fd1a3da445527907d13e09d2"} Oct 30 05:29:51.942456 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:51.942389 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" podStartSLOduration=1.942375299 podStartE2EDuration="1.942375299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:29:51.941287308 +0000 UTC m=+5515.227162084" watchObservedRunningTime="2026-10-30 05:29:51.942375299 +0000 UTC m=+5515.228250075" Oct 30 05:29:52.179783 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:52.179751 4793 patch_prober.go:29] interesting pod/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 05:29:52.179959 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:52.179844 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 05:29:52.323317 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:52.323270 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:29:53.181793 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:53.181748 4793 patch_prober.go:29] interesting pod/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body= Oct 30 05:29:53.182003 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:53.181814 4793 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" Oct 30 05:29:53.528094 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:53.528011 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:54.382791 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:54.382759 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:54.383281 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:54.383251 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:54.383947 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:54.383928 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:29:55.431738 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:55.431702 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:57.155223 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:57.155185 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:29:58.949273 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:29:58.949242 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:00.143304 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.143270 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px"] Oct 30 05:30:00.145935 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.145914 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" Oct 30 05:30:00.165448 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.165404 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px"] Oct 30 05:30:00.173542 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.173518 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:30:00.173657 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.173542 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:30:00.173657 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.173518 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:30:00.182338 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.182317 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s52wn\" (UniqueName: \"kubernetes.io/projected/574add30-c869-40c0-8548-601da00b920b-kube-api-access-s52wn\") pod \"maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px\" (UID: \"574add30-c869-40c0-8548-601da00b920b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" Oct 30 05:30:00.182463 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.182369 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/574add30-c869-40c0-8548-601da00b920b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px\" (UID: \"574add30-c869-40c0-8548-601da00b920b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" Oct 30 05:30:00.283898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.283859 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/574add30-c869-40c0-8548-601da00b920b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px\" (UID: \"574add30-c869-40c0-8548-601da00b920b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" Oct 30 05:30:00.284112 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.283982 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s52wn\" (UniqueName: \"kubernetes.io/projected/574add30-c869-40c0-8548-601da00b920b-kube-api-access-s52wn\") pod \"maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px\" (UID: \"574add30-c869-40c0-8548-601da00b920b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" Oct 30 05:30:00.284568 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.284546 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/574add30-c869-40c0-8548-601da00b920b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px\" (UID: \"574add30-c869-40c0-8548-601da00b920b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" Oct 30 05:30:00.307455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.307409 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s52wn\" (UniqueName: \"kubernetes.io/projected/574add30-c869-40c0-8548-601da00b920b-kube-api-access-s52wn\") pod \"maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px\" (UID: \"574add30-c869-40c0-8548-601da00b920b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" Oct 30 05:30:00.459278 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.459249 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" Oct 30 05:30:00.739818 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.739739 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:00.755969 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.755937 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px"] Oct 30 05:30:00.757137 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:30:00.757104 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574add30_c869_40c0_8548_601da00b920b.slice/crio-81491e40bae2559bcfb8bd5a5fa66acaaa267d3a350a2ff2092dda297f789e4c WatchSource:0}: Error finding container 81491e40bae2559bcfb8bd5a5fa66acaaa267d3a350a2ff2092dda297f789e4c: Status 404 returned error can't find the container with id 81491e40bae2559bcfb8bd5a5fa66acaaa267d3a350a2ff2092dda297f789e4c Oct 30 05:30:00.953019 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:00.952984 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" event={"ID":"574add30-c869-40c0-8548-601da00b920b","Type":"ContainerStarted","Data":"81491e40bae2559bcfb8bd5a5fa66acaaa267d3a350a2ff2092dda297f789e4c"} Oct 30 05:30:01.324004 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:01.323929 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:30:01.336696 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:01.336672 4793 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:30:01.606914 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:01.606851 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:01.960579 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:01.960546 4793 generic.go:356] "Generic (PLEG): container finished" podID="574add30-c869-40c0-8548-601da00b920b" containerID="527d9789129368b0a834e00f8cd3ff22c70a01cee786bcf3f61788d19e67ae18" exitCode=60 Oct 30 05:30:01.960780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:01.960625 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" event={"ID":"574add30-c869-40c0-8548-601da00b920b","Type":"ContainerDied","Data":"527d9789129368b0a834e00f8cd3ff22c70a01cee786bcf3f61788d19e67ae18"} Oct 30 05:30:01.960853 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:01.960817 4793 scope.go:122] "RemoveContainer" containerID="527d9789129368b0a834e00f8cd3ff22c70a01cee786bcf3f61788d19e67ae18" Oct 30 05:30:01.972837 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:01.972817 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:30:02.639603 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:02.639502 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:02.968204 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:02.968167 4793 generic.go:356] "Generic (PLEG): container finished" podID="574add30-c869-40c0-8548-601da00b920b" containerID="1f1f3694464147183da906280680d3390d647311898d3f9a8051a2184a783807" exitCode=60 Oct 30 05:30:02.968387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:02.968198 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" event={"ID":"574add30-c869-40c0-8548-601da00b920b","Type":"ContainerDied","Data":"1f1f3694464147183da906280680d3390d647311898d3f9a8051a2184a783807"} Oct 30 05:30:02.968387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:02.968268 4793 scope.go:122] "RemoveContainer" containerID="527d9789129368b0a834e00f8cd3ff22c70a01cee786bcf3f61788d19e67ae18" Oct 30 05:30:02.968523 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:02.968489 4793 scope.go:122] "RemoveContainer" containerID="1f1f3694464147183da906280680d3390d647311898d3f9a8051a2184a783807" Oct 30 05:30:02.968773 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:30:02.968750 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px_odh-ai-gateway-infra(574add30-c869-40c0-8548-601da00b920b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" podUID="574add30-c869-40c0-8548-601da00b920b" Oct 30 05:30:03.975903 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:03.975876 4793 scope.go:122] "RemoveContainer" containerID="1f1f3694464147183da906280680d3390d647311898d3f9a8051a2184a783807" Oct 30 05:30:03.976298 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:30:03.976069 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px_odh-ai-gateway-infra(574add30-c869-40c0-8548-601da00b920b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" podUID="574add30-c869-40c0-8548-601da00b920b" Oct 30 05:30:04.468486 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:04.468440 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:06.466290 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:06.466257 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:08.139285 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:08.139251 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:09.032455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:09.032409 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b"] Oct 30 05:30:09.044117 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:09.044089 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-xjm2b"] Oct 30 05:30:09.967461 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:09.967345 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:10.844578 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:10.844549 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fca5e27-9fda-45d7-83f7-fd0d25147bd1" path="/var/lib/kubelet/pods/5fca5e27-9fda-45d7-83f7-fd0d25147bd1/volumes" Oct 30 05:30:11.218180 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:30:11.218145 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:30:11.858683 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:11.858648 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:13.738266 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:13.738223 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:15.546380 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:15.546342 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:15.839177 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:15.839101 4793 scope.go:122] "RemoveContainer" containerID="1f1f3694464147183da906280680d3390d647311898d3f9a8051a2184a783807" Oct 30 05:30:17.055814 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:17.055776 4793 generic.go:356] "Generic (PLEG): container finished" podID="574add30-c869-40c0-8548-601da00b920b" containerID="0cb282ae3a478bb7d193ea084bb29f2880aa167fc3b33fbd80e85a742bd1a60a" exitCode=60 Oct 30 05:30:17.056261 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:17.055848 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" event={"ID":"574add30-c869-40c0-8548-601da00b920b","Type":"ContainerDied","Data":"0cb282ae3a478bb7d193ea084bb29f2880aa167fc3b33fbd80e85a742bd1a60a"} Oct 30 05:30:17.056261 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:17.055905 4793 scope.go:122] "RemoveContainer" containerID="1f1f3694464147183da906280680d3390d647311898d3f9a8051a2184a783807" Oct 30 05:30:17.056261 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:17.056186 4793 scope.go:122] "RemoveContainer" containerID="0cb282ae3a478bb7d193ea084bb29f2880aa167fc3b33fbd80e85a742bd1a60a" Oct 30 05:30:17.056477 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:30:17.056391 4793 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px_odh-ai-gateway-infra(574add30-c869-40c0-8548-601da00b920b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" podUID="574add30-c869-40c0-8548-601da00b920b" Oct 30 05:30:17.456995 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:17.456958 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:18.109067 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:18.109033 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px"] Oct 30 05:30:18.255761 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:18.255735 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" Oct 30 05:30:18.364765 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:18.364690 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/574add30-c869-40c0-8548-601da00b920b-kube-api-access-s52wn\" (UniqueName: \"kubernetes.io/projected/574add30-c869-40c0-8548-601da00b920b-kube-api-access-s52wn\") pod \"574add30-c869-40c0-8548-601da00b920b\" (UID: \"574add30-c869-40c0-8548-601da00b920b\") " Oct 30 05:30:18.364765 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:18.364736 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/574add30-c869-40c0-8548-601da00b920b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/574add30-c869-40c0-8548-601da00b920b-maas-api-service-ca\") pod \"574add30-c869-40c0-8548-601da00b920b\" (UID: \"574add30-c869-40c0-8548-601da00b920b\") " Oct 30 05:30:18.365193 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:18.365150 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/574add30-c869-40c0-8548-601da00b920b-maas-api-service-ca" pod "574add30-c869-40c0-8548-601da00b920b" (UID: "574add30-c869-40c0-8548-601da00b920b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:18.367181 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:18.367137 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/574add30-c869-40c0-8548-601da00b920b-kube-api-access-s52wn" pod "574add30-c869-40c0-8548-601da00b920b" (UID: "574add30-c869-40c0-8548-601da00b920b"). InnerVolumeSpecName "kube-api-access-s52wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:18.465857 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:18.465827 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s52wn\" (UniqueName: \"kubernetes.io/projected/574add30-c869-40c0-8548-601da00b920b-kube-api-access-s52wn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:18.465857 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:18.465851 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/574add30-c869-40c0-8548-601da00b920b-maas-api-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:19.069775 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:19.069745 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" Oct 30 05:30:19.069961 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:19.069793 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px" event={"ID":"574add30-c869-40c0-8548-601da00b920b","Type":"ContainerDied","Data":"81491e40bae2559bcfb8bd5a5fa66acaaa267d3a350a2ff2092dda297f789e4c"} Oct 30 05:30:19.069961 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:19.069861 4793 scope.go:122] "RemoveContainer" containerID="0cb282ae3a478bb7d193ea084bb29f2880aa167fc3b33fbd80e85a742bd1a60a" Oct 30 05:30:19.125513 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:19.125459 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px"] Oct 30 05:30:19.137197 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:19.137175 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-042f4c-29888978z4px"] Oct 30 05:30:19.341024 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:19.340978 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:20.844430 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:20.844392 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="574add30-c869-40c0-8548-601da00b920b" path="/var/lib/kubelet/pods/574add30-c869-40c0-8548-601da00b920b/volumes" Oct 30 05:30:21.147903 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:21.147809 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:22.233524 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.233492 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v"] Oct 30 05:30:22.233933 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.233868 4793 cpu_manager.go:414] "containerMap: removing container" podUID="574add30-c869-40c0-8548-601da00b920b" containerName="cleanup" Oct 30 05:30:22.233933 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.233880 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="574add30-c869-40c0-8548-601da00b920b" containerName="cleanup" Oct 30 05:30:22.233933 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.233899 4793 cpu_manager.go:414] "containerMap: removing container" podUID="574add30-c869-40c0-8548-601da00b920b" containerName="cleanup" Oct 30 05:30:22.233933 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.233904 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="574add30-c869-40c0-8548-601da00b920b" containerName="cleanup" Oct 30 05:30:22.234099 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.233972 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="574add30-c869-40c0-8548-601da00b920b" containerName="cleanup" Oct 30 05:30:22.234099 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.233982 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="574add30-c869-40c0-8548-601da00b920b" containerName="cleanup" Oct 30 05:30:22.234099 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.233989 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="574add30-c869-40c0-8548-601da00b920b" containerName="cleanup" Oct 30 05:30:22.238584 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.238566 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:22.256140 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.256111 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v"] Oct 30 05:30:22.264642 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.264607 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:30:22.264642 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.264636 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:30:22.264841 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.264652 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:30:22.264841 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.264729 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:30:22.404258 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.404219 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a4e1d5-96j2v\" (UID: \"ed75a697-f2ed-4a0c-9284-53e431081891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:22.404464 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.404274 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a4e1d5-96j2v\" (UID: \"ed75a697-f2ed-4a0c-9284-53e431081891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:22.404464 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.404297 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxqzn\" (UniqueName: \"kubernetes.io/projected/ed75a697-f2ed-4a0c-9284-53e431081891-kube-api-access-cxqzn\") pod \"maas-api-revoke-sub-93a4e1d5-96j2v\" (UID: \"ed75a697-f2ed-4a0c-9284-53e431081891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:22.505494 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.505432 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a4e1d5-96j2v\" (UID: \"ed75a697-f2ed-4a0c-9284-53e431081891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:22.505494 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.505483 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a4e1d5-96j2v\" (UID: \"ed75a697-f2ed-4a0c-9284-53e431081891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:22.505639 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.505505 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxqzn\" (UniqueName: \"kubernetes.io/projected/ed75a697-f2ed-4a0c-9284-53e431081891-kube-api-access-cxqzn\") pod \"maas-api-revoke-sub-93a4e1d5-96j2v\" (UID: \"ed75a697-f2ed-4a0c-9284-53e431081891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:22.506137 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.506116 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a4e1d5-96j2v\" (UID: \"ed75a697-f2ed-4a0c-9284-53e431081891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:22.507698 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.507680 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a4e1d5-96j2v\" (UID: \"ed75a697-f2ed-4a0c-9284-53e431081891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:22.528598 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.528579 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxqzn\" (UniqueName: \"kubernetes.io/projected/ed75a697-f2ed-4a0c-9284-53e431081891-kube-api-access-cxqzn\") pod \"maas-api-revoke-sub-93a4e1d5-96j2v\" (UID: \"ed75a697-f2ed-4a0c-9284-53e431081891\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:22.551455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.551406 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:22.847718 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:22.847691 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v"] Oct 30 05:30:22.848308 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:30:22.848277 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded75a697_f2ed_4a0c_9284_53e431081891.slice/crio-f0e4c48426377702879725b6c81cc77b42b77d04a88d91436d3b46cceb2ad18c WatchSource:0}: Error finding container f0e4c48426377702879725b6c81cc77b42b77d04a88d91436d3b46cceb2ad18c: Status 404 returned error can't find the container with id f0e4c48426377702879725b6c81cc77b42b77d04a88d91436d3b46cceb2ad18c Oct 30 05:30:23.038566 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:23.038529 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:23.099491 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:23.099409 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" event={"ID":"ed75a697-f2ed-4a0c-9284-53e431081891","Type":"ContainerStarted","Data":"f0e4c48426377702879725b6c81cc77b42b77d04a88d91436d3b46cceb2ad18c"} Oct 30 05:30:24.107497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:24.107465 4793 generic.go:356] "Generic (PLEG): container finished" podID="ed75a697-f2ed-4a0c-9284-53e431081891" containerID="3992b822fc27b7d6c3234486764ee69abf7e386dc4f01346c073b883bc4ce102" exitCode=0 Oct 30 05:30:24.107885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:24.107509 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" event={"ID":"ed75a697-f2ed-4a0c-9284-53e431081891","Type":"ContainerDied","Data":"3992b822fc27b7d6c3234486764ee69abf7e386dc4f01346c073b883bc4ce102"} Oct 30 05:30:24.843234 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:24.843165 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:25.248946 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:25.248925 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:25.331036 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:25.331005 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed75a697-f2ed-4a0c-9284-53e431081891-kube-api-access-cxqzn\" (UniqueName: \"kubernetes.io/projected/ed75a697-f2ed-4a0c-9284-53e431081891-kube-api-access-cxqzn\") pod \"ed75a697-f2ed-4a0c-9284-53e431081891\" (UID: \"ed75a697-f2ed-4a0c-9284-53e431081891\") " Oct 30 05:30:25.331213 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:25.331066 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-xks-ca\") pod \"ed75a697-f2ed-4a0c-9284-53e431081891\" (UID: \"ed75a697-f2ed-4a0c-9284-53e431081891\") " Oct 30 05:30:25.331213 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:25.331100 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-openshift-ca\") pod \"ed75a697-f2ed-4a0c-9284-53e431081891\" (UID: \"ed75a697-f2ed-4a0c-9284-53e431081891\") " Oct 30 05:30:25.331509 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:25.331481 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-openshift-ca" pod "ed75a697-f2ed-4a0c-9284-53e431081891" (UID: "ed75a697-f2ed-4a0c-9284-53e431081891"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:25.333031 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:25.333009 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-xks-ca" pod "ed75a697-f2ed-4a0c-9284-53e431081891" (UID: "ed75a697-f2ed-4a0c-9284-53e431081891"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:30:25.333136 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:25.333030 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed75a697-f2ed-4a0c-9284-53e431081891-kube-api-access-cxqzn" pod "ed75a697-f2ed-4a0c-9284-53e431081891" (UID: "ed75a697-f2ed-4a0c-9284-53e431081891"). InnerVolumeSpecName "kube-api-access-cxqzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:25.432458 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:25.432409 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxqzn\" (UniqueName: \"kubernetes.io/projected/ed75a697-f2ed-4a0c-9284-53e431081891-kube-api-access-cxqzn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:25.432568 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:25.432459 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:25.432568 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:25.432469 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed75a697-f2ed-4a0c-9284-53e431081891-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:26.120217 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:26.120186 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" Oct 30 05:30:26.120405 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:26.120231 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v" event={"ID":"ed75a697-f2ed-4a0c-9284-53e431081891","Type":"ContainerDied","Data":"f0e4c48426377702879725b6c81cc77b42b77d04a88d91436d3b46cceb2ad18c"} Oct 30 05:30:26.120405 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:26.120288 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e4c48426377702879725b6c81cc77b42b77d04a88d91436d3b46cceb2ad18c" Oct 30 05:30:26.651497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:26.651455 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:28.509070 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:28.509029 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:30.347563 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:30.347524 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:32.147780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:32.147745 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:33.475091 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:33.475055 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:33.942865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:33.942832 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:34.031093 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:34.031063 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc"] Oct 30 05:30:34.041326 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:34.041295 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-rspsc"] Oct 30 05:30:34.158110 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:34.158074 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:34.853347 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:34.853307 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aafded1c-d4cb-4209-ba55-52a4f55a0000" path="/var/lib/kubelet/pods/aafded1c-d4cb-4209-ba55-52a4f55a0000/volumes" Oct 30 05:30:35.889184 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:35.889149 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:37.660491 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:37.660453 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:38.294715 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:38.294683 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8"] Oct 30 05:30:38.501873 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:38.501834 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:39.247438 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:39.247392 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82"] Oct 30 05:30:39.486094 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:39.486058 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:39.724204 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:39.724173 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:40.931072 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:40.931032 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:41.215464 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:41.215349 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:41.218801 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:30:41.218774 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:30:44.248483 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:44.248397 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" podUID="986e35e1-95ca-44a3-8218-ffa8ae132fb3" containerName="payload-pre-processing" containerID="cri-o://e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616" gracePeriod=25 Oct 30 05:30:46.973402 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:46.973364 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:47.195122 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:47.195076 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:47.716331 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:47.716287 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:47.793387 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:47.793347 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" podUID="986e35e1-95ca-44a3-8218-ffa8ae132fb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:30:47.998517 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:47.998434 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn"] Oct 30 05:30:47.999004 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:47.998658 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" containerID="cri-o://6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411" gracePeriod=30 Oct 30 05:30:48.137517 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:48.137475 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:30:49.246328 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:49.246291 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp"] Oct 30 05:30:49.257246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:49.257216 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-88bpp"] Oct 30 05:30:50.845972 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:50.845934 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbbec7af-1642-4f51-b128-187b2a078846" path="/var/lib/kubelet/pods/fbbec7af-1642-4f51-b128-187b2a078846/volumes" Oct 30 05:30:53.246197 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.246174 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:30:53.281460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.281373 4793 generic.go:356] "Generic (PLEG): container finished" podID="43fce029-53ab-4b52-b743-13803d84ffef" containerID="6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411" exitCode=0 Oct 30 05:30:53.281460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.281438 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" Oct 30 05:30:53.281603 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.281462 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" event={"ID":"43fce029-53ab-4b52-b743-13803d84ffef","Type":"ContainerDied","Data":"6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411"} Oct 30 05:30:53.281603 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.281516 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn" event={"ID":"43fce029-53ab-4b52-b743-13803d84ffef","Type":"ContainerDied","Data":"18efe97c09f3b6c9fe7f68d38b96a2cb027879fea16dceb67a3b5b6e671e6e84"} Oct 30 05:30:53.281603 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.281539 4793 scope.go:122] "RemoveContainer" containerID="6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411" Oct 30 05:30:53.290662 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.290645 4793 scope.go:122] "RemoveContainer" containerID="7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164" Oct 30 05:30:53.300548 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.300458 4793 scope.go:122] "RemoveContainer" containerID="6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411" Oct 30 05:30:53.301037 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:30:53.301007 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411\": container with ID starting with 6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411 not found: ID does not exist" containerID="6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411" Oct 30 05:30:53.301129 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.301049 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411"} err="failed to get container status \"6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411\": rpc error: code = NotFound desc = could not find container \"6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411\": container with ID starting with 6db294d1d67bca7b9fabbf9aa84ee5f95f3e173148a901a33a9a59fef5faf411 not found: ID does not exist" Oct 30 05:30:53.301129 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.301067 4793 scope.go:122] "RemoveContainer" containerID="7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164" Oct 30 05:30:53.301340 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:30:53.301322 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164\": container with ID starting with 7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164 not found: ID does not exist" containerID="7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164" Oct 30 05:30:53.301393 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.301346 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164"} err="failed to get container status \"7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164\": rpc error: code = NotFound desc = could not find container \"7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164\": container with ID starting with 7592f0f369a210504ceda1bde97d37be060d98d56081a58d6a55afbfcfead164 not found: ID does not exist" Oct 30 05:30:53.314746 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.314717 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" podUID="e999f18e-33dd-4050-b3ed-48ae27eed876" containerName="main" containerID="cri-o://4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6" gracePeriod=45 Oct 30 05:30:53.315714 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.315691 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-kube-api-access-xlc7j\" (UniqueName: \"kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-kube-api-access-xlc7j\") pod \"43fce029-53ab-4b52-b743-13803d84ffef\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " Oct 30 05:30:53.315824 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.315743 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-socket\") pod \"43fce029-53ab-4b52-b743-13803d84ffef\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " Oct 30 05:30:53.315824 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.315798 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-credential-socket\") pod \"43fce029-53ab-4b52-b743-13803d84ffef\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " Oct 30 05:30:53.315950 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.315846 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-istio-token\" (UniqueName: \"kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-istio-token\") pod \"43fce029-53ab-4b52-b743-13803d84ffef\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " Oct 30 05:30:53.315950 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.315894 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43fce029-53ab-4b52-b743-13803d84ffef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43fce029-53ab-4b52-b743-13803d84ffef-istiod-ca-cert\") pod \"43fce029-53ab-4b52-b743-13803d84ffef\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " Oct 30 05:30:53.315950 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.315933 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-data\") pod \"43fce029-53ab-4b52-b743-13803d84ffef\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " Oct 30 05:30:53.316105 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.316022 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-certs\") pod \"43fce029-53ab-4b52-b743-13803d84ffef\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " Oct 30 05:30:53.316105 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.316066 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43fce029-53ab-4b52-b743-13803d84ffef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43fce029-53ab-4b52-b743-13803d84ffef-istio-podinfo\") pod \"43fce029-53ab-4b52-b743-13803d84ffef\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " Oct 30 05:30:53.316105 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.316093 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-envoy\") pod \"43fce029-53ab-4b52-b743-13803d84ffef\" (UID: \"43fce029-53ab-4b52-b743-13803d84ffef\") " Oct 30 05:30:53.316818 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.316786 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-data" pod "43fce029-53ab-4b52-b743-13803d84ffef" (UID: "43fce029-53ab-4b52-b743-13803d84ffef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:53.317276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.317028 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-certs" pod "43fce029-53ab-4b52-b743-13803d84ffef" (UID: "43fce029-53ab-4b52-b743-13803d84ffef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:53.317829 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.317149 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43fce029-53ab-4b52-b743-13803d84ffef-istiod-ca-cert" pod "43fce029-53ab-4b52-b743-13803d84ffef" (UID: "43fce029-53ab-4b52-b743-13803d84ffef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:30:53.318555 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.318520 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-credential-socket" pod "43fce029-53ab-4b52-b743-13803d84ffef" (UID: "43fce029-53ab-4b52-b743-13803d84ffef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:53.318755 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.318718 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-socket" pod "43fce029-53ab-4b52-b743-13803d84ffef" (UID: "43fce029-53ab-4b52-b743-13803d84ffef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:53.319228 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.319203 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-kube-api-access-xlc7j" pod "43fce029-53ab-4b52-b743-13803d84ffef" (UID: "43fce029-53ab-4b52-b743-13803d84ffef"). InnerVolumeSpecName "kube-api-access-xlc7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:53.319693 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.319648 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-istio-token" pod "43fce029-53ab-4b52-b743-13803d84ffef" (UID: "43fce029-53ab-4b52-b743-13803d84ffef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:53.321485 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.320686 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-envoy" pod "43fce029-53ab-4b52-b743-13803d84ffef" (UID: "43fce029-53ab-4b52-b743-13803d84ffef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:53.333590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.333562 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43fce029-53ab-4b52-b743-13803d84ffef-istio-podinfo" pod "43fce029-53ab-4b52-b743-13803d84ffef" (UID: "43fce029-53ab-4b52-b743-13803d84ffef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:30:53.417386 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.417363 4793 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-credential-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.417386 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.417387 4793 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-istio-token\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.417552 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.417396 4793 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43fce029-53ab-4b52-b743-13803d84ffef-istiod-ca-cert\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.417552 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.417404 4793 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-data\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.417552 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.417412 4793 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.417552 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.417444 4793 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43fce029-53ab-4b52-b743-13803d84ffef-istio-podinfo\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.417552 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.417451 4793 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-istio-envoy\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.417552 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.417461 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlc7j\" (UniqueName: \"kubernetes.io/projected/43fce029-53ab-4b52-b743-13803d84ffef-kube-api-access-xlc7j\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.417552 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.417469 4793 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43fce029-53ab-4b52-b743-13803d84ffef-workload-socket\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.554782 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.554759 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:30:53.633434 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.633390 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn"] Oct 30 05:30:53.645397 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.645367 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-042f4c-openshift-default-54d9f7fc7c98rpn"] Oct 30 05:30:53.720680 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.720646 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-dshm\") pod \"e999f18e-33dd-4050-b3ed-48ae27eed876\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " Oct 30 05:30:53.720865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.720736 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-home\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-home\") pod \"e999f18e-33dd-4050-b3ed-48ae27eed876\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " Oct 30 05:30:53.720865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.720762 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e999f18e-33dd-4050-b3ed-48ae27eed876-kube-api-access-l8zpn\" (UniqueName: \"kubernetes.io/projected/e999f18e-33dd-4050-b3ed-48ae27eed876-kube-api-access-l8zpn\") pod \"e999f18e-33dd-4050-b3ed-48ae27eed876\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " Oct 30 05:30:53.720865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.720786 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e999f18e-33dd-4050-b3ed-48ae27eed876-tls-certs\" (UniqueName: \"kubernetes.io/secret/e999f18e-33dd-4050-b3ed-48ae27eed876-tls-certs\") pod \"e999f18e-33dd-4050-b3ed-48ae27eed876\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " Oct 30 05:30:53.720865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.720807 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-tmp-dir\") pod \"e999f18e-33dd-4050-b3ed-48ae27eed876\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " Oct 30 05:30:53.720865 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.720847 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-model-cache\") pod \"e999f18e-33dd-4050-b3ed-48ae27eed876\" (UID: \"e999f18e-33dd-4050-b3ed-48ae27eed876\") " Oct 30 05:30:53.721176 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.721085 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-home" pod "e999f18e-33dd-4050-b3ed-48ae27eed876" (UID: "e999f18e-33dd-4050-b3ed-48ae27eed876"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:53.721176 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.721114 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-tmp-dir" pod "e999f18e-33dd-4050-b3ed-48ae27eed876" (UID: "e999f18e-33dd-4050-b3ed-48ae27eed876"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:53.721321 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.721293 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-model-cache" pod "e999f18e-33dd-4050-b3ed-48ae27eed876" (UID: "e999f18e-33dd-4050-b3ed-48ae27eed876"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:53.721400 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.721383 4793 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-home\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.721484 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.721408 4793 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-tmp-dir\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.723014 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.722992 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-dshm" pod "e999f18e-33dd-4050-b3ed-48ae27eed876" (UID: "e999f18e-33dd-4050-b3ed-48ae27eed876"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:30:53.723145 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.723115 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e999f18e-33dd-4050-b3ed-48ae27eed876-tls-certs" pod "e999f18e-33dd-4050-b3ed-48ae27eed876" (UID: "e999f18e-33dd-4050-b3ed-48ae27eed876"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:30:53.723203 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.723120 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e999f18e-33dd-4050-b3ed-48ae27eed876-kube-api-access-l8zpn" pod "e999f18e-33dd-4050-b3ed-48ae27eed876" (UID: "e999f18e-33dd-4050-b3ed-48ae27eed876"). InnerVolumeSpecName "kube-api-access-l8zpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:30:53.822683 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.822615 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8zpn\" (UniqueName: \"kubernetes.io/projected/e999f18e-33dd-4050-b3ed-48ae27eed876-kube-api-access-l8zpn\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.822683 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.822640 4793 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e999f18e-33dd-4050-b3ed-48ae27eed876-tls-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.822683 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.822656 4793 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-model-cache\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:53.822683 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:53.822668 4793 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e999f18e-33dd-4050-b3ed-48ae27eed876-dshm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:30:54.028912 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.028886 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68"] Oct 30 05:30:54.032735 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.032710 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-lln68"] Oct 30 05:30:54.289439 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.289382 4793 generic.go:356] "Generic (PLEG): container finished" podID="e999f18e-33dd-4050-b3ed-48ae27eed876" containerID="4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6" exitCode=0 Oct 30 05:30:54.289898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.289450 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" Oct 30 05:30:54.289898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.289454 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" event={"ID":"e999f18e-33dd-4050-b3ed-48ae27eed876","Type":"ContainerDied","Data":"4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6"} Oct 30 05:30:54.289898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.289497 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8" event={"ID":"e999f18e-33dd-4050-b3ed-48ae27eed876","Type":"ContainerDied","Data":"86832abec47641cfe41908140ebae9d7bc7d5b11fd1a3da445527907d13e09d2"} Oct 30 05:30:54.289898 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.289513 4793 scope.go:122] "RemoveContainer" containerID="4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6" Oct 30 05:30:54.299347 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.299328 4793 scope.go:122] "RemoveContainer" containerID="4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6" Oct 30 05:30:54.299669 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:30:54.299647 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6\": container with ID starting with 4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6 not found: ID does not exist" containerID="4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6" Oct 30 05:30:54.299776 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.299674 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6"} err="failed to get container status \"4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6\": rpc error: code = NotFound desc = could not find container \"4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6\": container with ID starting with 4ad2922dbd792068dd53d6deb82df7d73bdf112809bf3d899c7aa95a814758a6 not found: ID does not exist" Oct 30 05:30:54.339913 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.339889 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8"] Oct 30 05:30:54.351398 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.351371 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-042f4c/simulator-single-main-042f4c-kserve-57ff664d95-fwsb8"] Oct 30 05:30:54.845556 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.845526 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43fce029-53ab-4b52-b743-13803d84ffef" path="/var/lib/kubelet/pods/43fce029-53ab-4b52-b743-13803d84ffef/volumes" Oct 30 05:30:54.845944 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.845930 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5" path="/var/lib/kubelet/pods/87c4fbca-0f9f-40a8-adfb-c59e6f8ac7e5/volumes" Oct 30 05:30:54.846319 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:54.846307 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e999f18e-33dd-4050-b3ed-48ae27eed876" path="/var/lib/kubelet/pods/e999f18e-33dd-4050-b3ed-48ae27eed876/volumes" Oct 30 05:30:57.792922 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:30:57.792884 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" podUID="986e35e1-95ca-44a3-8218-ffa8ae132fb3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:31:01.735808 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:01.735775 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:31:04.500124 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:04.500095 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" Oct 30 05:31:04.554565 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:04.554490 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/986e35e1-95ca-44a3-8218-ffa8ae132fb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986e35e1-95ca-44a3-8218-ffa8ae132fb3-plugins-config-volume\") pod \"986e35e1-95ca-44a3-8218-ffa8ae132fb3\" (UID: \"986e35e1-95ca-44a3-8218-ffa8ae132fb3\") " Oct 30 05:31:04.554726 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:04.554597 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/986e35e1-95ca-44a3-8218-ffa8ae132fb3-kube-api-access-brm8n\" (UniqueName: \"kubernetes.io/projected/986e35e1-95ca-44a3-8218-ffa8ae132fb3-kube-api-access-brm8n\") pod \"986e35e1-95ca-44a3-8218-ffa8ae132fb3\" (UID: \"986e35e1-95ca-44a3-8218-ffa8ae132fb3\") " Oct 30 05:31:04.554918 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:04.554895 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986e35e1-95ca-44a3-8218-ffa8ae132fb3-plugins-config-volume" pod "986e35e1-95ca-44a3-8218-ffa8ae132fb3" (UID: "986e35e1-95ca-44a3-8218-ffa8ae132fb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:04.554992 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:04.554969 4793 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/986e35e1-95ca-44a3-8218-ffa8ae132fb3-plugins-config-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:31:04.556757 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:04.556731 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986e35e1-95ca-44a3-8218-ffa8ae132fb3-kube-api-access-brm8n" pod "986e35e1-95ca-44a3-8218-ffa8ae132fb3" (UID: "986e35e1-95ca-44a3-8218-ffa8ae132fb3"). InnerVolumeSpecName "kube-api-access-brm8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:04.655605 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:04.655574 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brm8n\" (UniqueName: \"kubernetes.io/projected/986e35e1-95ca-44a3-8218-ffa8ae132fb3-kube-api-access-brm8n\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:31:05.032236 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:05.032199 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g"] Oct 30 05:31:05.035617 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:05.035596 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-44n7g"] Oct 30 05:31:05.361006 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:05.360965 4793 generic.go:356] "Generic (PLEG): container finished" podID="986e35e1-95ca-44a3-8218-ffa8ae132fb3" containerID="e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616" exitCode=0 Oct 30 05:31:05.361006 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:05.361009 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" Oct 30 05:31:05.361274 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:05.361051 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" event={"ID":"986e35e1-95ca-44a3-8218-ffa8ae132fb3","Type":"ContainerDied","Data":"e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616"} Oct 30 05:31:05.361274 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:05.361109 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82" event={"ID":"986e35e1-95ca-44a3-8218-ffa8ae132fb3","Type":"ContainerDied","Data":"84727b81747a65cf1c0f99bf4b3a0a640750260dc27a76ec56c74e6d7463c5fa"} Oct 30 05:31:05.361274 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:05.361142 4793 scope.go:122] "RemoveContainer" containerID="e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616" Oct 30 05:31:05.371650 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:05.371362 4793 scope.go:122] "RemoveContainer" containerID="e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616" Oct 30 05:31:05.371728 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:31:05.371675 4793 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616\": container with ID starting with e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616 not found: ID does not exist" containerID="e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616" Oct 30 05:31:05.371728 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:05.371697 4793 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616"} err="failed to get container status \"e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616\": rpc error: code = NotFound desc = could not find container \"e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616\": container with ID starting with e6590fd6a41b5f0d66de362df11e1137a445ace0350686a396db3c8359c7e616 not found: ID does not exist" Oct 30 05:31:05.416950 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:05.416923 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82"] Oct 30 05:31:05.421615 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:05.421592 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-042f4c-9d7ccjrh82"] Oct 30 05:31:06.845723 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:06.845689 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43fddb59-b066-45d5-8cd3-a516d90b8ee2" path="/var/lib/kubelet/pods/43fddb59-b066-45d5-8cd3-a516d90b8ee2/volumes" Oct 30 05:31:06.846116 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:06.846082 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="986e35e1-95ca-44a3-8218-ffa8ae132fb3" path="/var/lib/kubelet/pods/986e35e1-95ca-44a3-8218-ffa8ae132fb3/volumes" Oct 30 05:31:07.172375 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:07.172297 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg"] Oct 30 05:31:07.172561 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:07.172517 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" podUID="9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793" containerName="manager" containerID="cri-o://488d38c78ed8fe2f4e6f00a62f67b44c23fcccfc1093543e4475c9c97d9be3fe" gracePeriod=10 Oct 30 05:31:07.375670 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:07.375633 4793 generic.go:356] "Generic (PLEG): container finished" podID="9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793" containerID="488d38c78ed8fe2f4e6f00a62f67b44c23fcccfc1093543e4475c9c97d9be3fe" exitCode=0 Oct 30 05:31:07.375809 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:07.375690 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" event={"ID":"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793","Type":"ContainerDied","Data":"488d38c78ed8fe2f4e6f00a62f67b44c23fcccfc1093543e4475c9c97d9be3fe"} Oct 30 05:31:07.422933 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:07.422883 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" Oct 30 05:31:07.481078 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:07.481053 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-kube-api-access-zhkqc\" (UniqueName: \"kubernetes.io/projected/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-kube-api-access-zhkqc\") pod \"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793\" (UID: \"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793\") " Oct 30 05:31:07.481605 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:07.481206 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-extensions-socket-volume\") pod \"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793\" (UID: \"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793\") " Oct 30 05:31:07.484220 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:07.484160 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-kube-api-access-zhkqc" pod "9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793" (UID: "9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793"). InnerVolumeSpecName "kube-api-access-zhkqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:07.493383 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:07.492992 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-extensions-socket-volume" pod "9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793" (UID: "9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:07.583318 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:07.583282 4793 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-extensions-socket-volume\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:31:07.583318 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:07.583323 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhkqc\" (UniqueName: \"kubernetes.io/projected/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793-kube-api-access-zhkqc\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:31:08.382861 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:08.382829 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" Oct 30 05:31:08.383281 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:08.382835 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg" event={"ID":"9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793","Type":"ContainerDied","Data":"af056a9df554968938b268cfd9794f45e64b8ba4df9390c814824cec4c22a0a5"} Oct 30 05:31:08.383281 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:08.382964 4793 scope.go:122] "RemoveContainer" containerID="488d38c78ed8fe2f4e6f00a62f67b44c23fcccfc1093543e4475c9c97d9be3fe" Oct 30 05:31:08.450113 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:08.450084 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg"] Oct 30 05:31:08.461748 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:08.461721 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcmbg"] Oct 30 05:31:08.844581 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:08.844547 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793" path="/var/lib/kubelet/pods/9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793/volumes" Oct 30 05:31:11.218565 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:31:11.218532 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:31:16.158194 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:16.158154 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:31:16.420196 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:16.420113 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:31:22.085271 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:22.085234 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:31:33.030240 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:33.030203 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq"] Oct 30 05:31:33.041556 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:33.041530 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kl5mq"] Oct 30 05:31:34.843615 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:34.843583 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dfebbac-4dcf-424f-b612-7a152d071d74" path="/var/lib/kubelet/pods/4dfebbac-4dcf-424f-b612-7a152d071d74/volumes" Oct 30 05:31:41.218080 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:31:41.217999 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:31:44.030467 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:44.030437 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv"] Oct 30 05:31:44.041033 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:44.041009 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-652lv"] Oct 30 05:31:44.843815 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:31:44.843781 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9dd94e-0cf7-47c9-b0d5-6946929d91a1" path="/var/lib/kubelet/pods/1d9dd94e-0cf7-47c9-b0d5-6946929d91a1/volumes" Oct 30 05:32:00.219265 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:00.219230 4793 scope.go:122] "RemoveContainer" containerID="b021553431f8d24e8cfbe4ad823ba72f61a57cd2d9227ffce79c7fb0234044df" Oct 30 05:32:00.228139 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:00.228118 4793 scope.go:122] "RemoveContainer" containerID="82960618a8d8e21230c730a1e4314b78209bac60bb922a63cd1b54b54cd09d0a" Oct 30 05:32:00.236145 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:00.236127 4793 scope.go:122] "RemoveContainer" containerID="110035b61b488c0bab0263ce5c0db3217ab9fd2b349aea3003baf9224f2c5ae6" Oct 30 05:32:01.853106 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:01.853071 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:32:05.028246 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:05.028212 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d"] Oct 30 05:32:05.039397 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:05.039371 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-f5g5d"] Oct 30 05:32:06.844130 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:06.844098 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260b9450-132e-42ea-a1a4-987c65627c08" path="/var/lib/kubelet/pods/260b9450-132e-42ea-a1a4-987c65627c08/volumes" Oct 30 05:32:10.292587 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.292552 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2"] Oct 30 05:32:10.292973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.292907 4793 cpu_manager.go:414] "containerMap: removing container" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" Oct 30 05:32:10.292973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.292918 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" Oct 30 05:32:10.292973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.292930 4793 cpu_manager.go:414] "containerMap: removing container" podUID="574add30-c869-40c0-8548-601da00b920b" containerName="cleanup" Oct 30 05:32:10.292973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.292935 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="574add30-c869-40c0-8548-601da00b920b" containerName="cleanup" Oct 30 05:32:10.292973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.292948 4793 cpu_manager.go:414] "containerMap: removing container" podUID="e999f18e-33dd-4050-b3ed-48ae27eed876" containerName="main" Oct 30 05:32:10.292973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.292954 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e999f18e-33dd-4050-b3ed-48ae27eed876" containerName="main" Oct 30 05:32:10.292973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.292961 4793 cpu_manager.go:414] "containerMap: removing container" podUID="ed75a697-f2ed-4a0c-9284-53e431081891" containerName="revoke-keys" Oct 30 05:32:10.292973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.292966 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed75a697-f2ed-4a0c-9284-53e431081891" containerName="revoke-keys" Oct 30 05:32:10.293276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.292983 4793 cpu_manager.go:414] "containerMap: removing container" podUID="9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793" containerName="manager" Oct 30 05:32:10.293276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.292988 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793" containerName="manager" Oct 30 05:32:10.293276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.292995 4793 cpu_manager.go:414] "containerMap: removing container" podUID="986e35e1-95ca-44a3-8218-ffa8ae132fb3" containerName="payload-pre-processing" Oct 30 05:32:10.293276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.293000 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="986e35e1-95ca-44a3-8218-ffa8ae132fb3" containerName="payload-pre-processing" Oct 30 05:32:10.293276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.293059 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="e999f18e-33dd-4050-b3ed-48ae27eed876" containerName="main" Oct 30 05:32:10.293276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.293066 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed75a697-f2ed-4a0c-9284-53e431081891" containerName="revoke-keys" Oct 30 05:32:10.293276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.293075 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7da71d-ab0a-44a7-bcc6-b0dd09f7f793" containerName="manager" Oct 30 05:32:10.293276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.293081 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" Oct 30 05:32:10.293276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.293086 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="986e35e1-95ca-44a3-8218-ffa8ae132fb3" containerName="payload-pre-processing" Oct 30 05:32:10.293276 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.293093 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" Oct 30 05:32:10.297850 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.297829 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" Oct 30 05:32:10.309460 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.309433 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2"] Oct 30 05:32:10.317846 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.317823 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wmr7g\"" Oct 30 05:32:10.405031 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.405003 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7975f468-e137-4d99-8a6d-ad9ac46c8961-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7nhv2\" (UID: \"7975f468-e137-4d99-8a6d-ad9ac46c8961\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" Oct 30 05:32:10.405166 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.405055 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hntkj\" (UniqueName: \"kubernetes.io/projected/7975f468-e137-4d99-8a6d-ad9ac46c8961-kube-api-access-hntkj\") pod \"kuadrant-operator-controller-manager-678d87766f-7nhv2\" (UID: \"7975f468-e137-4d99-8a6d-ad9ac46c8961\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" Oct 30 05:32:10.506239 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.506211 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hntkj\" (UniqueName: \"kubernetes.io/projected/7975f468-e137-4d99-8a6d-ad9ac46c8961-kube-api-access-hntkj\") pod \"kuadrant-operator-controller-manager-678d87766f-7nhv2\" (UID: \"7975f468-e137-4d99-8a6d-ad9ac46c8961\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" Oct 30 05:32:10.506360 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.506279 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7975f468-e137-4d99-8a6d-ad9ac46c8961-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7nhv2\" (UID: \"7975f468-e137-4d99-8a6d-ad9ac46c8961\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" Oct 30 05:32:10.506642 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.506626 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7975f468-e137-4d99-8a6d-ad9ac46c8961-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7nhv2\" (UID: \"7975f468-e137-4d99-8a6d-ad9ac46c8961\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" Oct 30 05:32:10.530532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.530504 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hntkj\" (UniqueName: \"kubernetes.io/projected/7975f468-e137-4d99-8a6d-ad9ac46c8961-kube-api-access-hntkj\") pod \"kuadrant-operator-controller-manager-678d87766f-7nhv2\" (UID: \"7975f468-e137-4d99-8a6d-ad9ac46c8961\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" Oct 30 05:32:10.612229 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.612175 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" Oct 30 05:32:10.937553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:10.937522 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2"] Oct 30 05:32:10.950325 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:32:10.950299 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7975f468_e137_4d99_8a6d_ad9ac46c8961.slice/crio-3fdf55dfa68ba711c79e23e18ca79e9dfc16ccdaf0213f3ec80b9363fa706dbe WatchSource:0}: Error finding container 3fdf55dfa68ba711c79e23e18ca79e9dfc16ccdaf0213f3ec80b9363fa706dbe: Status 404 returned error can't find the container with id 3fdf55dfa68ba711c79e23e18ca79e9dfc16ccdaf0213f3ec80b9363fa706dbe Oct 30 05:32:11.218594 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:32:11.218517 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:32:11.740429 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:11.740390 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" event={"ID":"7975f468-e137-4d99-8a6d-ad9ac46c8961","Type":"ContainerStarted","Data":"5669d8ff4f0160dd622f1afb6d99ba7473b88cd7913ce575ebf893f1d6e2ee11"} Oct 30 05:32:11.740822 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:11.740459 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" event={"ID":"7975f468-e137-4d99-8a6d-ad9ac46c8961","Type":"ContainerStarted","Data":"3fdf55dfa68ba711c79e23e18ca79e9dfc16ccdaf0213f3ec80b9363fa706dbe"} Oct 30 05:32:11.740822 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:11.740482 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" Oct 30 05:32:11.809009 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:11.808960 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" podStartSLOduration=1.8089476740000001 podStartE2EDuration="1.808947674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:11.794629153 +0000 UTC m=+5655.080503928" watchObservedRunningTime="2026-10-30 05:32:11.808947674 +0000 UTC m=+5655.094822428" Oct 30 05:32:21.035753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:21.035722 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt"] Oct 30 05:32:21.038465 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:21.038446 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-74wzt"] Oct 30 05:32:22.527338 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:22.527301 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:32:22.748597 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:22.748571 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7nhv2" Oct 30 05:32:22.752374 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:22.752346 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:32:22.846278 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:22.846200 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e93211f-42e5-4c75-a2fe-55f0cb874b47" path="/var/lib/kubelet/pods/5e93211f-42e5-4c75-a2fe-55f0cb874b47/volumes" Oct 30 05:32:35.010134 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.010103 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs"] Oct 30 05:32:35.010549 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.010497 4793 cpu_manager.go:414] "containerMap: removing container" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" Oct 30 05:32:35.010549 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.010509 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43fce029-53ab-4b52-b743-13803d84ffef" containerName="istio-proxy" Oct 30 05:32:35.013836 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.013818 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:35.042399 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.042375 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:32:35.042540 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.042455 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:32:35.045144 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.045125 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs"] Oct 30 05:32:35.050884 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.050866 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:32:35.050971 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.050895 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:32:35.155657 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.155624 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846be85a-3645-4604-a284-5f439aaaf632-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xthrs\" (UID: \"846be85a-3645-4604-a284-5f439aaaf632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:35.155778 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.155704 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846be85a-3645-4604-a284-5f439aaaf632-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xthrs\" (UID: \"846be85a-3645-4604-a284-5f439aaaf632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:35.155831 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.155786 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgp2r\" (UniqueName: \"kubernetes.io/projected/846be85a-3645-4604-a284-5f439aaaf632-kube-api-access-dgp2r\") pod \"maas-api-revoke-sub-d576e68b-xthrs\" (UID: \"846be85a-3645-4604-a284-5f439aaaf632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:35.256601 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.256567 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846be85a-3645-4604-a284-5f439aaaf632-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xthrs\" (UID: \"846be85a-3645-4604-a284-5f439aaaf632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:35.256752 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.256640 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846be85a-3645-4604-a284-5f439aaaf632-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xthrs\" (UID: \"846be85a-3645-4604-a284-5f439aaaf632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:35.256752 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.256670 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgp2r\" (UniqueName: \"kubernetes.io/projected/846be85a-3645-4604-a284-5f439aaaf632-kube-api-access-dgp2r\") pod \"maas-api-revoke-sub-d576e68b-xthrs\" (UID: \"846be85a-3645-4604-a284-5f439aaaf632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:35.257258 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.257234 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846be85a-3645-4604-a284-5f439aaaf632-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-xthrs\" (UID: \"846be85a-3645-4604-a284-5f439aaaf632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:35.258973 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.258951 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846be85a-3645-4604-a284-5f439aaaf632-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-xthrs\" (UID: \"846be85a-3645-4604-a284-5f439aaaf632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:35.284192 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.284138 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgp2r\" (UniqueName: \"kubernetes.io/projected/846be85a-3645-4604-a284-5f439aaaf632-kube-api-access-dgp2r\") pod \"maas-api-revoke-sub-d576e68b-xthrs\" (UID: \"846be85a-3645-4604-a284-5f439aaaf632\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:35.326786 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.326760 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:35.628183 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.628148 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs"] Oct 30 05:32:35.629860 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:32:35.629831 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846be85a_3645_4604_a284_5f439aaaf632.slice/crio-be1deccff103207d5109c52c7db35a069703d827da82be9a50c278bcc578bc89 WatchSource:0}: Error finding container be1deccff103207d5109c52c7db35a069703d827da82be9a50c278bcc578bc89: Status 404 returned error can't find the container with id be1deccff103207d5109c52c7db35a069703d827da82be9a50c278bcc578bc89 Oct 30 05:32:35.964253 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:35.964222 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" event={"ID":"846be85a-3645-4604-a284-5f439aaaf632","Type":"ContainerStarted","Data":"be1deccff103207d5109c52c7db35a069703d827da82be9a50c278bcc578bc89"} Oct 30 05:32:36.970740 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:36.970707 4793 generic.go:356] "Generic (PLEG): container finished" podID="846be85a-3645-4604-a284-5f439aaaf632" containerID="6ca6f3bafec6c4bbdff0357928f12879e6d7d013d4e3e8132c2f4f0c49844b41" exitCode=0 Oct 30 05:32:36.971149 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:36.970794 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" event={"ID":"846be85a-3645-4604-a284-5f439aaaf632","Type":"ContainerDied","Data":"6ca6f3bafec6c4bbdff0357928f12879e6d7d013d4e3e8132c2f4f0c49844b41"} Oct 30 05:32:38.033377 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.033342 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb"] Oct 30 05:32:38.044773 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.044747 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-jc5kb"] Oct 30 05:32:38.107579 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.107557 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:38.180166 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.180135 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/846be85a-3645-4604-a284-5f439aaaf632-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846be85a-3645-4604-a284-5f439aaaf632-maas-api-openshift-ca\") pod \"846be85a-3645-4604-a284-5f439aaaf632\" (UID: \"846be85a-3645-4604-a284-5f439aaaf632\") " Oct 30 05:32:38.180282 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.180179 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/846be85a-3645-4604-a284-5f439aaaf632-kube-api-access-dgp2r\" (UniqueName: \"kubernetes.io/projected/846be85a-3645-4604-a284-5f439aaaf632-kube-api-access-dgp2r\") pod \"846be85a-3645-4604-a284-5f439aaaf632\" (UID: \"846be85a-3645-4604-a284-5f439aaaf632\") " Oct 30 05:32:38.180325 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.180287 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/846be85a-3645-4604-a284-5f439aaaf632-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846be85a-3645-4604-a284-5f439aaaf632-maas-api-xks-ca\") pod \"846be85a-3645-4604-a284-5f439aaaf632\" (UID: \"846be85a-3645-4604-a284-5f439aaaf632\") " Oct 30 05:32:38.180558 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.180534 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846be85a-3645-4604-a284-5f439aaaf632-maas-api-openshift-ca" pod "846be85a-3645-4604-a284-5f439aaaf632" (UID: "846be85a-3645-4604-a284-5f439aaaf632"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:38.180668 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.180653 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/846be85a-3645-4604-a284-5f439aaaf632-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:32:38.182379 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.182363 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846be85a-3645-4604-a284-5f439aaaf632-kube-api-access-dgp2r" pod "846be85a-3645-4604-a284-5f439aaaf632" (UID: "846be85a-3645-4604-a284-5f439aaaf632"). InnerVolumeSpecName "kube-api-access-dgp2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:38.182455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.182367 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/846be85a-3645-4604-a284-5f439aaaf632-maas-api-xks-ca" pod "846be85a-3645-4604-a284-5f439aaaf632" (UID: "846be85a-3645-4604-a284-5f439aaaf632"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:38.281264 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.281209 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgp2r\" (UniqueName: \"kubernetes.io/projected/846be85a-3645-4604-a284-5f439aaaf632-kube-api-access-dgp2r\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:32:38.281264 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.281232 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/846be85a-3645-4604-a284-5f439aaaf632-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:32:38.843144 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.843112 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d7e926-b408-4f0e-b13a-a629ed6a6421" path="/var/lib/kubelet/pods/24d7e926-b408-4f0e-b13a-a629ed6a6421/volumes" Oct 30 05:32:38.982580 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.982549 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" Oct 30 05:32:38.982738 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.982579 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs" event={"ID":"846be85a-3645-4604-a284-5f439aaaf632","Type":"ContainerDied","Data":"be1deccff103207d5109c52c7db35a069703d827da82be9a50c278bcc578bc89"} Oct 30 05:32:38.982738 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:38.982654 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be1deccff103207d5109c52c7db35a069703d827da82be9a50c278bcc578bc89" Oct 30 05:32:41.217915 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:32:41.217878 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:32:50.065050 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.065011 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78"] Oct 30 05:32:50.065648 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.065564 4793 cpu_manager.go:414] "containerMap: removing container" podUID="846be85a-3645-4604-a284-5f439aaaf632" containerName="revoke-keys" Oct 30 05:32:50.065648 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.065582 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="846be85a-3645-4604-a284-5f439aaaf632" containerName="revoke-keys" Oct 30 05:32:50.065789 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.065710 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="846be85a-3645-4604-a284-5f439aaaf632" containerName="revoke-keys" Oct 30 05:32:50.069169 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.069148 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:50.095370 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.095119 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:32:50.095370 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.095222 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:32:50.095370 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.095336 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4lg8c\"" Oct 30 05:32:50.095774 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.095752 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:32:50.099596 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.099568 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78"] Oct 30 05:32:50.183177 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.183133 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ghm78\" (UID: \"19203538-0e22-4d4d-a078-d41692e14f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:50.183389 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.183190 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ghm78\" (UID: \"19203538-0e22-4d4d-a078-d41692e14f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:50.183389 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.183314 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qcdm\" (UniqueName: \"kubernetes.io/projected/19203538-0e22-4d4d-a078-d41692e14f20-kube-api-access-9qcdm\") pod \"maas-api-revoke-sub-d2bc93bc-ghm78\" (UID: \"19203538-0e22-4d4d-a078-d41692e14f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:50.284356 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.284314 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qcdm\" (UniqueName: \"kubernetes.io/projected/19203538-0e22-4d4d-a078-d41692e14f20-kube-api-access-9qcdm\") pod \"maas-api-revoke-sub-d2bc93bc-ghm78\" (UID: \"19203538-0e22-4d4d-a078-d41692e14f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:50.285121 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.285090 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ghm78\" (UID: \"19203538-0e22-4d4d-a078-d41692e14f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:50.286213 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.286191 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ghm78\" (UID: \"19203538-0e22-4d4d-a078-d41692e14f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:50.286834 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.286132 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ghm78\" (UID: \"19203538-0e22-4d4d-a078-d41692e14f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:50.289832 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.289807 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-ghm78\" (UID: \"19203538-0e22-4d4d-a078-d41692e14f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:50.319448 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.319356 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qcdm\" (UniqueName: \"kubernetes.io/projected/19203538-0e22-4d4d-a078-d41692e14f20-kube-api-access-9qcdm\") pod \"maas-api-revoke-sub-d2bc93bc-ghm78\" (UID: \"19203538-0e22-4d4d-a078-d41692e14f20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:50.387642 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.387611 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:50.995059 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:50.995024 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78"] Oct 30 05:32:51.104140 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:51.103980 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" event={"ID":"19203538-0e22-4d4d-a078-d41692e14f20","Type":"ContainerStarted","Data":"5c6d320f4dfa200e82efb0cf852541bb9b1f831813c4804f22182fb67b2cc056"} Oct 30 05:32:52.034449 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:52.033961 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9"] Oct 30 05:32:52.044916 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:52.044889 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-4nzb9"] Oct 30 05:32:52.117566 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:52.117520 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" event={"ID":"19203538-0e22-4d4d-a078-d41692e14f20","Type":"ContainerStarted","Data":"09e911a795671ecd4caeaa94966b5beae96777be0b6e98d9eab4881ee1b03408"} Oct 30 05:32:52.159832 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:52.159781 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" podStartSLOduration=2.159762918 podStartE2EDuration="2.159762918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:52.157026094 +0000 UTC m=+5695.442900881" watchObservedRunningTime="2026-10-30 05:32:52.159762918 +0000 UTC m=+5695.445637694" Oct 30 05:32:52.844553 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:52.844481 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0595324-4f00-471e-8acc-b498ce5dd5f6" path="/var/lib/kubelet/pods/d0595324-4f00-471e-8acc-b498ce5dd5f6/volumes" Oct 30 05:32:53.125789 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:53.125715 4793 generic.go:356] "Generic (PLEG): container finished" podID="19203538-0e22-4d4d-a078-d41692e14f20" containerID="09e911a795671ecd4caeaa94966b5beae96777be0b6e98d9eab4881ee1b03408" exitCode=0 Oct 30 05:32:53.125789 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:53.125771 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" event={"ID":"19203538-0e22-4d4d-a078-d41692e14f20","Type":"ContainerDied","Data":"09e911a795671ecd4caeaa94966b5beae96777be0b6e98d9eab4881ee1b03408"} Oct 30 05:32:54.272607 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:54.272585 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:54.369817 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:54.369783 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19203538-0e22-4d4d-a078-d41692e14f20-kube-api-access-9qcdm\" (UniqueName: \"kubernetes.io/projected/19203538-0e22-4d4d-a078-d41692e14f20-kube-api-access-9qcdm\") pod \"19203538-0e22-4d4d-a078-d41692e14f20\" (UID: \"19203538-0e22-4d4d-a078-d41692e14f20\") " Oct 30 05:32:54.369972 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:54.369856 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-xks-ca\") pod \"19203538-0e22-4d4d-a078-d41692e14f20\" (UID: \"19203538-0e22-4d4d-a078-d41692e14f20\") " Oct 30 05:32:54.369972 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:54.369941 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-openshift-ca\") pod \"19203538-0e22-4d4d-a078-d41692e14f20\" (UID: \"19203538-0e22-4d4d-a078-d41692e14f20\") " Oct 30 05:32:54.370311 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:54.370285 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-openshift-ca" pod "19203538-0e22-4d4d-a078-d41692e14f20" (UID: "19203538-0e22-4d4d-a078-d41692e14f20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:54.372160 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:54.372130 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19203538-0e22-4d4d-a078-d41692e14f20-kube-api-access-9qcdm" pod "19203538-0e22-4d4d-a078-d41692e14f20" (UID: "19203538-0e22-4d4d-a078-d41692e14f20"). InnerVolumeSpecName "kube-api-access-9qcdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:54.372297 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:54.372156 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-xks-ca" pod "19203538-0e22-4d4d-a078-d41692e14f20" (UID: "19203538-0e22-4d4d-a078-d41692e14f20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:54.471198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:54.471170 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-xks-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:32:54.471198 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:54.471196 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19203538-0e22-4d4d-a078-d41692e14f20-maas-api-openshift-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:32:54.471377 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:54.471208 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qcdm\" (UniqueName: \"kubernetes.io/projected/19203538-0e22-4d4d-a078-d41692e14f20-kube-api-access-9qcdm\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:32:55.138131 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:55.138039 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" event={"ID":"19203538-0e22-4d4d-a078-d41692e14f20","Type":"ContainerDied","Data":"5c6d320f4dfa200e82efb0cf852541bb9b1f831813c4804f22182fb67b2cc056"} Oct 30 05:32:55.138131 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:55.138079 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78" Oct 30 05:32:55.138131 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:32:55.138098 4793 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c6d320f4dfa200e82efb0cf852541bb9b1f831813c4804f22182fb67b2cc056" Oct 30 05:33:00.273888 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:00.273860 4793 scope.go:122] "RemoveContainer" containerID="99f3c7ebf74accee8b0c3e62061c58b7050ea034663c1d9f2fe0f24b9c2ae9fa" Oct 30 05:33:00.284201 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:00.284180 4793 scope.go:122] "RemoveContainer" containerID="cc1d8870e9383687c31c4f19b3c148841f088fc696d8b6b9385b6df9c490814d" Oct 30 05:33:00.292326 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:00.292305 4793 scope.go:122] "RemoveContainer" containerID="6f125db57c9fdb6bf0812b7ea4b30e607ab46ad71d8e07fb621bd4ea68276578" Oct 30 05:33:01.970499 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:01.970458 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:33:02.149971 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.149941 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c"] Oct 30 05:33:02.150374 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.150356 4793 cpu_manager.go:414] "containerMap: removing container" podUID="19203538-0e22-4d4d-a078-d41692e14f20" containerName="revoke-keys" Oct 30 05:33:02.150374 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.150373 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19203538-0e22-4d4d-a078-d41692e14f20" containerName="revoke-keys" Oct 30 05:33:02.150569 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.150484 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="19203538-0e22-4d4d-a078-d41692e14f20" containerName="revoke-keys" Oct 30 05:33:02.155304 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.155281 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.184845 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.184761 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:33:02.185120 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.185099 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 05:33:02.185247 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.185184 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:33:02.185322 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.185261 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gw5bk\"" Oct 30 05:33:02.185541 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.185515 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 05:33:02.187958 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.187933 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c"] Oct 30 05:33:02.242170 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.242100 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5096e530-d491-4cdd-bac1-af21627f41a4-openshift-service-ca\") pod \"maas-api-58d7f7c4f4-hfl5c\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.242301 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.242213 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-maas-api-tls\") pod \"maas-api-58d7f7c4f4-hfl5c\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.242301 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.242268 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-metrics-certs\") pod \"maas-api-58d7f7c4f4-hfl5c\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.242380 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.242303 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqhnj\" (UniqueName: \"kubernetes.io/projected/5096e530-d491-4cdd-bac1-af21627f41a4-kube-api-access-jqhnj\") pod \"maas-api-58d7f7c4f4-hfl5c\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.343286 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.343260 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5096e530-d491-4cdd-bac1-af21627f41a4-openshift-service-ca\") pod \"maas-api-58d7f7c4f4-hfl5c\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.343488 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.343327 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-maas-api-tls\") pod \"maas-api-58d7f7c4f4-hfl5c\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.343488 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.343367 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-metrics-certs\") pod \"maas-api-58d7f7c4f4-hfl5c\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.343488 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.343410 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqhnj\" (UniqueName: \"kubernetes.io/projected/5096e530-d491-4cdd-bac1-af21627f41a4-kube-api-access-jqhnj\") pod \"maas-api-58d7f7c4f4-hfl5c\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.343989 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.343966 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5096e530-d491-4cdd-bac1-af21627f41a4-openshift-service-ca\") pod \"maas-api-58d7f7c4f4-hfl5c\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.345874 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.345854 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-maas-api-tls\") pod \"maas-api-58d7f7c4f4-hfl5c\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.345950 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.345897 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-metrics-certs\") pod \"maas-api-58d7f7c4f4-hfl5c\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.366464 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.366440 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqhnj\" (UniqueName: \"kubernetes.io/projected/5096e530-d491-4cdd-bac1-af21627f41a4-kube-api-access-jqhnj\") pod \"maas-api-58d7f7c4f4-hfl5c\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.471227 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.471202 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:02.772815 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.772791 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c"] Oct 30 05:33:02.774676 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:33:02.774650 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5096e530_d491_4cdd_bac1_af21627f41a4.slice/crio-fb0edb6cf5661d591360aab092c523d6433e831b376ba4de61ebdbe5316d878c WatchSource:0}: Error finding container fb0edb6cf5661d591360aab092c523d6433e831b376ba4de61ebdbe5316d878c: Status 404 returned error can't find the container with id fb0edb6cf5661d591360aab092c523d6433e831b376ba4de61ebdbe5316d878c Oct 30 05:33:02.776016 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:02.775999 4793 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:33:03.192890 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:03.192861 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" event={"ID":"5096e530-d491-4cdd-bac1-af21627f41a4","Type":"ContainerStarted","Data":"fb0edb6cf5661d591360aab092c523d6433e831b376ba4de61ebdbe5316d878c"} Oct 30 05:33:04.200106 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:04.200073 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" event={"ID":"5096e530-d491-4cdd-bac1-af21627f41a4","Type":"ContainerStarted","Data":"39d463b960cc0ad2750dd53306db2cca2592f369d177fe4fe7c736ae86f03e72"} Oct 30 05:33:04.200628 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:04.200197 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:04.243684 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:04.243638 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" podStartSLOduration=1.864501902 podStartE2EDuration="2.243624585s" totalImagesPullingTime="379.122683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:04.242928193 +0000 UTC m=+5707.528802970" watchObservedRunningTime="2026-10-30 05:33:04.243624585 +0000 UTC m=+5707.529499361" Oct 30 05:33:10.034126 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.034034 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c"] Oct 30 05:33:10.034613 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.034277 4793 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" podUID="5096e530-d491-4cdd-bac1-af21627f41a4" containerName="maas-api" containerID="cri-o://39d463b960cc0ad2750dd53306db2cca2592f369d177fe4fe7c736ae86f03e72" gracePeriod=30 Oct 30 05:33:10.038857 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.038836 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:10.102468 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.102434 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh"] Oct 30 05:33:10.105885 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.105865 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.133809 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.133783 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh"] Oct 30 05:33:10.218021 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.217988 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45adc9d8-925d-4d61-af90-9b6b5b71dc20-maas-api-tls\") pod \"maas-api-7b94bfd958-tdffh\" (UID: \"45adc9d8-925d-4d61-af90-9b6b5b71dc20\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.218169 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.218152 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45adc9d8-925d-4d61-af90-9b6b5b71dc20-metrics-certs\") pod \"maas-api-7b94bfd958-tdffh\" (UID: \"45adc9d8-925d-4d61-af90-9b6b5b71dc20\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.218235 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.218218 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45adc9d8-925d-4d61-af90-9b6b5b71dc20-openshift-service-ca\") pod \"maas-api-7b94bfd958-tdffh\" (UID: \"45adc9d8-925d-4d61-af90-9b6b5b71dc20\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.218349 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.218329 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gktwq\" (UniqueName: \"kubernetes.io/projected/45adc9d8-925d-4d61-af90-9b6b5b71dc20-kube-api-access-gktwq\") pod \"maas-api-7b94bfd958-tdffh\" (UID: \"45adc9d8-925d-4d61-af90-9b6b5b71dc20\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.234989 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.234955 4793 generic.go:356] "Generic (PLEG): container finished" podID="5096e530-d491-4cdd-bac1-af21627f41a4" containerID="39d463b960cc0ad2750dd53306db2cca2592f369d177fe4fe7c736ae86f03e72" exitCode=0 Oct 30 05:33:10.235106 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.235029 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" event={"ID":"5096e530-d491-4cdd-bac1-af21627f41a4","Type":"ContainerDied","Data":"39d463b960cc0ad2750dd53306db2cca2592f369d177fe4fe7c736ae86f03e72"} Oct 30 05:33:10.285856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.285804 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:10.318819 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.318788 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45adc9d8-925d-4d61-af90-9b6b5b71dc20-maas-api-tls\") pod \"maas-api-7b94bfd958-tdffh\" (UID: \"45adc9d8-925d-4d61-af90-9b6b5b71dc20\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.318938 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.318855 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45adc9d8-925d-4d61-af90-9b6b5b71dc20-metrics-certs\") pod \"maas-api-7b94bfd958-tdffh\" (UID: \"45adc9d8-925d-4d61-af90-9b6b5b71dc20\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.318938 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.318894 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45adc9d8-925d-4d61-af90-9b6b5b71dc20-openshift-service-ca\") pod \"maas-api-7b94bfd958-tdffh\" (UID: \"45adc9d8-925d-4d61-af90-9b6b5b71dc20\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.318938 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.318930 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gktwq\" (UniqueName: \"kubernetes.io/projected/45adc9d8-925d-4d61-af90-9b6b5b71dc20-kube-api-access-gktwq\") pod \"maas-api-7b94bfd958-tdffh\" (UID: \"45adc9d8-925d-4d61-af90-9b6b5b71dc20\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.319590 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.319566 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45adc9d8-925d-4d61-af90-9b6b5b71dc20-openshift-service-ca\") pod \"maas-api-7b94bfd958-tdffh\" (UID: \"45adc9d8-925d-4d61-af90-9b6b5b71dc20\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.321265 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.321242 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45adc9d8-925d-4d61-af90-9b6b5b71dc20-metrics-certs\") pod \"maas-api-7b94bfd958-tdffh\" (UID: \"45adc9d8-925d-4d61-af90-9b6b5b71dc20\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.321364 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.321307 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45adc9d8-925d-4d61-af90-9b6b5b71dc20-maas-api-tls\") pod \"maas-api-7b94bfd958-tdffh\" (UID: \"45adc9d8-925d-4d61-af90-9b6b5b71dc20\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.344312 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.344288 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gktwq\" (UniqueName: \"kubernetes.io/projected/45adc9d8-925d-4d61-af90-9b6b5b71dc20-kube-api-access-gktwq\") pod \"maas-api-7b94bfd958-tdffh\" (UID: \"45adc9d8-925d-4d61-af90-9b6b5b71dc20\") " pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.419375 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.419349 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5096e530-d491-4cdd-bac1-af21627f41a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5096e530-d491-4cdd-bac1-af21627f41a4-openshift-service-ca\") pod \"5096e530-d491-4cdd-bac1-af21627f41a4\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " Oct 30 05:33:10.419532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.419388 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5096e530-d491-4cdd-bac1-af21627f41a4-kube-api-access-jqhnj\" (UniqueName: \"kubernetes.io/projected/5096e530-d491-4cdd-bac1-af21627f41a4-kube-api-access-jqhnj\") pod \"5096e530-d491-4cdd-bac1-af21627f41a4\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " Oct 30 05:33:10.419532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.419472 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-metrics-certs\") pod \"5096e530-d491-4cdd-bac1-af21627f41a4\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " Oct 30 05:33:10.419628 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.419579 4793 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-maas-api-tls\") pod \"5096e530-d491-4cdd-bac1-af21627f41a4\" (UID: \"5096e530-d491-4cdd-bac1-af21627f41a4\") " Oct 30 05:33:10.419789 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.419769 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5096e530-d491-4cdd-bac1-af21627f41a4-openshift-service-ca" pod "5096e530-d491-4cdd-bac1-af21627f41a4" (UID: "5096e530-d491-4cdd-bac1-af21627f41a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:33:10.419856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.419840 4793 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5096e530-d491-4cdd-bac1-af21627f41a4-openshift-service-ca\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:33:10.421384 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.421357 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5096e530-d491-4cdd-bac1-af21627f41a4-kube-api-access-jqhnj" pod "5096e530-d491-4cdd-bac1-af21627f41a4" (UID: "5096e530-d491-4cdd-bac1-af21627f41a4"). InnerVolumeSpecName "kube-api-access-jqhnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:33:10.421579 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.421554 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-metrics-certs" pod "5096e530-d491-4cdd-bac1-af21627f41a4" (UID: "5096e530-d491-4cdd-bac1-af21627f41a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:10.421666 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.421586 4793 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-maas-api-tls" pod "5096e530-d491-4cdd-bac1-af21627f41a4" (UID: "5096e530-d491-4cdd-bac1-af21627f41a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:33:10.438753 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.438732 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:10.521145 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.521119 4793 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-maas-api-tls\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:33:10.521145 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.521143 4793 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqhnj\" (UniqueName: \"kubernetes.io/projected/5096e530-d491-4cdd-bac1-af21627f41a4-kube-api-access-jqhnj\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:33:10.521321 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.521154 4793 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5096e530-d491-4cdd-bac1-af21627f41a4-metrics-certs\") on node \"ip-10-0-138-177.ec2.internal\" DevicePath \"\"" Oct 30 05:33:10.741093 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:33:10.741053 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45adc9d8_925d_4d61_af90_9b6b5b71dc20.slice/crio-454a6bd9d8b6352971da3258c581f8c49ee078caaeef6288b670b66ba4539094 WatchSource:0}: Error finding container 454a6bd9d8b6352971da3258c581f8c49ee078caaeef6288b670b66ba4539094: Status 404 returned error can't find the container with id 454a6bd9d8b6352971da3258c581f8c49ee078caaeef6288b670b66ba4539094 Oct 30 05:33:10.741750 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:10.741724 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh"] Oct 30 05:33:11.218676 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:33:11.218636 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:33:11.242207 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:11.242182 4793 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" Oct 30 05:33:11.242207 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:11.242196 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c" event={"ID":"5096e530-d491-4cdd-bac1-af21627f41a4","Type":"ContainerDied","Data":"fb0edb6cf5661d591360aab092c523d6433e831b376ba4de61ebdbe5316d878c"} Oct 30 05:33:11.242402 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:11.242282 4793 scope.go:122] "RemoveContainer" containerID="39d463b960cc0ad2750dd53306db2cca2592f369d177fe4fe7c736ae86f03e72" Oct 30 05:33:11.243928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:11.243903 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" event={"ID":"45adc9d8-925d-4d61-af90-9b6b5b71dc20","Type":"ContainerStarted","Data":"0535391aa7f70a7ba522348fbf0772d40f863ee0a651a6f3706e9b5f4ef664b0"} Oct 30 05:33:11.244048 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:11.243942 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" event={"ID":"45adc9d8-925d-4d61-af90-9b6b5b71dc20","Type":"ContainerStarted","Data":"454a6bd9d8b6352971da3258c581f8c49ee078caaeef6288b670b66ba4539094"} Oct 30 05:33:11.244126 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:11.244046 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:11.293024 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:11.293000 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c"] Oct 30 05:33:11.304942 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:11.304916 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7f7c4f4-hfl5c"] Oct 30 05:33:11.340532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:11.340456 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" podStartSLOduration=0.99953864 podStartE2EDuration="1.340444453s" totalImagesPullingTime="340.905813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:33:11.337767012 +0000 UTC m=+5714.623641787" watchObservedRunningTime="2026-10-30 05:33:11.340444453 +0000 UTC m=+5714.626319230" Oct 30 05:33:12.844062 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:12.844022 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5096e530-d491-4cdd-bac1-af21627f41a4" path="/var/lib/kubelet/pods/5096e530-d491-4cdd-bac1-af21627f41a4/volumes" Oct 30 05:33:17.257131 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:17.257099 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b94bfd958-tdffh" Oct 30 05:33:20.450192 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:20.450161 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:33:41.218168 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:33:41.218134 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:33:51.030297 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:51.030253 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4"] Oct 30 05:33:51.041944 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:51.041907 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-szwq4"] Oct 30 05:33:52.845395 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:52.845357 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a820726d-168a-449c-9c87-6e290978a4cb" path="/var/lib/kubelet/pods/a820726d-168a-449c-9c87-6e290978a4cb/volumes" Oct 30 05:33:59.140484 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:59.140441 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:33:59.371492 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:33:59.371457 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:34:00.320917 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:00.320886 4793 scope.go:122] "RemoveContainer" containerID="2f8b7433d9b04e9aabfdd9301156a8e38389a85ed6c3be92a209f149cfd0ab08" Oct 30 05:34:00.329732 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:00.329714 4793 scope.go:122] "RemoveContainer" containerID="1e55df745334cc13872eda33e143fbd0d3ea43438abd4d3e55efb731c1497b6b" Oct 30 05:34:00.337779 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:00.337761 4793 scope.go:122] "RemoveContainer" containerID="4573d77d386069e5b81e7ff1d4500b19212869e603d2c0c8fa71661400fcb147" Oct 30 05:34:00.345177 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:00.345160 4793 scope.go:122] "RemoveContainer" containerID="b46d3e8d9273a26fcc586032567d604598926300ac9b65c71ca95f90acb47dd0" Oct 30 05:34:00.352389 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:00.352374 4793 scope.go:122] "RemoveContainer" containerID="b0f848bd6bed36f0afae67a52838177d51496e8c1d123428fe952bc9c7c8a7c5" Oct 30 05:34:01.027882 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:01.027853 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8"] Oct 30 05:34:01.039235 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:01.039208 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g8qw8"] Oct 30 05:34:02.099183 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:02.099143 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:34:02.843539 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:02.843504 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9" path="/var/lib/kubelet/pods/ae4cd8dc-bcc7-4a79-9e5e-1c98df955ca9/volumes" Oct 30 05:34:11.283443 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:34:11.280643 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:34:13.868915 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:13.868871 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:34:41.217849 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:34:41.217766 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:34:53.840884 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:53.840843 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mfjwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:34:53.843721 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:53.841484 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mfjwn" podUID="4a556af0-394a-4c79-a3b8-971c207edf59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:34:56.509455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:56.509180 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h7nkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:34:56.509455 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:34:56.509269 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podUID="84c295aa-6250-4bb0-bb4d-4fe24004a9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:35:00.391545 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:35:00.391505 4793 scope.go:122] "RemoveContainer" containerID="6844647d82d195c389ef43bfcda0022b0a861ed3c1742b337e04facc04eb3fdd" Oct 30 05:35:00.403208 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:35:00.403181 4793 scope.go:122] "RemoveContainer" containerID="1e739a8535a2a7b3f1406ad3f86427a42de5e7f7aafeaca76f9a3a5fb1ad7983" Oct 30 05:35:00.413728 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:35:00.413703 4793 scope.go:122] "RemoveContainer" containerID="eb65864ec19a14138bc7bfcfbab111de05159af5f005da403243dd8850948766" Oct 30 05:35:02.281089 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:35:02.281048 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:35:06.257169 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:35:06.257131 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:35:08.519931 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:35:08.519885 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:35:08.758661 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:35:08.758605 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:35:11.218366 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:35:11.218328 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:35:26.029183 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:35:26.029150 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v"] Oct 30 05:35:26.040532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:35:26.040503 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a4e1d5-96j2v"] Oct 30 05:35:26.845532 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:35:26.845499 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed75a697-f2ed-4a0c-9284-53e431081891" path="/var/lib/kubelet/pods/ed75a697-f2ed-4a0c-9284-53e431081891/volumes" Oct 30 05:35:41.224334 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:35:41.224271 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:36:00.450178 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:36:00.450146 4793 scope.go:122] "RemoveContainer" containerID="96e814daa50018829090bb73d26811c2ad417bcb04ce6efb533d359cc0c84101" Oct 30 05:36:02.394953 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:36:02.394905 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:36:11.218007 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:36:11.217926 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:36:41.218348 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:36:41.218310 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:37:00.472663 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:37:00.472632 4793 scope.go:122] "RemoveContainer" containerID="3992b822fc27b7d6c3234486764ee69abf7e386dc4f01346c073b883bc4ce102" Oct 30 05:37:02.523497 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:37:02.523452 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:37:11.218110 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:37:11.218071 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:37:26.398569 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:37:26.398530 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-c99g8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:37:26.399282 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:37:26.399247 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-c99g8" podUID="9b3ff486-e88f-459b-a0b1-aa5d88c6660d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:37:26.510563 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:37:26.510523 4793 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-h7nkg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:37:26.510864 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:37:26.510828 4793 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-h7nkg" podUID="84c295aa-6250-4bb0-bb4d-4fe24004a9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:37:39.031578 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:37:39.031547 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs"] Oct 30 05:37:39.042746 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:37:39.042722 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-xthrs"] Oct 30 05:37:40.846366 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:37:40.846293 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="846be85a-3645-4604-a284-5f439aaaf632" path="/var/lib/kubelet/pods/846be85a-3645-4604-a284-5f439aaaf632/volumes" Oct 30 05:37:41.218104 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:37:41.218069 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:37:55.029502 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:37:55.029466 4793 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78"] Oct 30 05:37:55.040879 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:37:55.040852 4793 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-ghm78"] Oct 30 05:37:56.903679 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:37:56.903633 4793 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19203538-0e22-4d4d-a078-d41692e14f20" path="/var/lib/kubelet/pods/19203538-0e22-4d4d-a078-d41692e14f20/volumes" Oct 30 05:38:02.782185 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:02.782138 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:38:11.218461 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:38:11.218398 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:38:16.191867 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:16.191832 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79d77bcfdb-6hwnl_a736cf03-a95d-4e46-9358-7b8e6bba483b/manager/0.log" Oct 30 05:38:17.651978 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:17.651939 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pxxpk_66328127-fc0b-4f8a-9599-2a82c297712d/manager/0.log" Oct 30 05:38:19.061928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:19.061896 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7b94bfd958-tdffh_45adc9d8-925d-4d61-af90-9b6b5b71dc20/maas-api/0.log" Oct 30 05:38:24.956214 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:24.956180 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-c4zkg_8af2e200-3999-4f9b-a10c-1407bef727e3/manager/0.log" Oct 30 05:38:25.150039 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:25.150009 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pg5vv_12b6686c-fc28-4cc0-9c53-252a4e8a02bd/manager/0.log" Oct 30 05:38:25.569987 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:25.569955 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7nhv2_7975f468-e137-4d99-8a6d-ad9ac46c8961/manager/0.log" Oct 30 05:38:26.961337 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:26.961308 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2_1179efa6-bb34-43ae-ab5d-ae9e3ec68028/istio-proxy/0.log" Oct 30 05:38:27.752953 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:27.752923 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-c99g8_9b3ff486-e88f-459b-a0b1-aa5d88c6660d/istio-proxy/0.log" Oct 30 05:38:28.405113 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:28.405083 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-h7nkg_84c295aa-6250-4bb0-bb4d-4fe24004a9f4/istio-proxy/0.log" Oct 30 05:38:29.017523 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:29.017489 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-mfjwn_4a556af0-394a-4c79-a3b8-971c207edf59/istio-proxy/0.log" Oct 30 05:38:29.839593 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:29.839560 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66_990707cc-7db4-490b-a520-9af03daeeeff/istio-proxy/0.log" Oct 30 05:38:30.198507 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:30.198472 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5688bd49d7-w75hb_e4815527-3e97-4f0b-bd06-be6277725976/payload-processing/0.log" Oct 30 05:38:31.356795 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:31.356763 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-df9pk_ce835b60-df8c-4ee0-a2a8-e3bb44b37b2b/main/0.log" Oct 30 05:38:31.554956 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:31.554925 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-xxxsx_05fb589e-1545-4aba-a706-ee3532bbfe65/main/0.log" Oct 30 05:38:31.958214 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:31.958176 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-wpknt_bac0e332-6063-4c8c-9445-0f3bd4946caf/main/0.log" Oct 30 05:38:32.543661 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:32.543634 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-b56rr_b4bb1e95-2059-43dd-8abf-384abf8cdefb/main/0.log" Oct 30 05:38:32.728177 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:32.728112 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-wqdvp_5097d128-40d6-4c8e-9b0b-6269985b6bed/main/0.log" Oct 30 05:38:41.217294 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:38:41.217258 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:38:43.455094 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.455056 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zwm6c/must-gather-2fspb"] Oct 30 05:38:43.455620 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.455469 4793 cpu_manager.go:414] "containerMap: removing container" podUID="5096e530-d491-4cdd-bac1-af21627f41a4" containerName="maas-api" Oct 30 05:38:43.455620 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.455492 4793 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5096e530-d491-4cdd-bac1-af21627f41a4" containerName="maas-api" Oct 30 05:38:43.455620 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.455556 4793 memory_manager.go:365] "RemoveStaleState removing state" podUID="5096e530-d491-4cdd-bac1-af21627f41a4" containerName="maas-api" Oct 30 05:38:43.459019 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.458998 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zwm6c/must-gather-2fspb" Oct 30 05:38:43.484522 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.484497 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zwm6c/must-gather-2fspb"] Oct 30 05:38:43.493396 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.493373 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zwm6c\"/\"kube-root-ca.crt\"" Oct 30 05:38:43.493516 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.493495 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zwm6c\"/\"default-dockercfg-rgptc\"" Oct 30 05:38:43.493566 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.493505 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zwm6c\"/\"openshift-service-ca.crt\"" Oct 30 05:38:43.556763 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.556731 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:38:43.568459 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.568435 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c4308aa5-f445-46c1-9103-80d42b64ce59-must-gather-output\") pod \"must-gather-2fspb\" (UID: \"c4308aa5-f445-46c1-9103-80d42b64ce59\") " pod="openshift-must-gather-zwm6c/must-gather-2fspb" Oct 30 05:38:43.568607 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.568516 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5md92\" (UniqueName: \"kubernetes.io/projected/c4308aa5-f445-46c1-9103-80d42b64ce59-kube-api-access-5md92\") pod \"must-gather-2fspb\" (UID: \"c4308aa5-f445-46c1-9103-80d42b64ce59\") " pod="openshift-must-gather-zwm6c/must-gather-2fspb" Oct 30 05:38:43.669480 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.669458 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5md92\" (UniqueName: \"kubernetes.io/projected/c4308aa5-f445-46c1-9103-80d42b64ce59-kube-api-access-5md92\") pod \"must-gather-2fspb\" (UID: \"c4308aa5-f445-46c1-9103-80d42b64ce59\") " pod="openshift-must-gather-zwm6c/must-gather-2fspb" Oct 30 05:38:43.669605 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.669516 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c4308aa5-f445-46c1-9103-80d42b64ce59-must-gather-output\") pod \"must-gather-2fspb\" (UID: \"c4308aa5-f445-46c1-9103-80d42b64ce59\") " pod="openshift-must-gather-zwm6c/must-gather-2fspb" Oct 30 05:38:43.669810 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.669792 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c4308aa5-f445-46c1-9103-80d42b64ce59-must-gather-output\") pod \"must-gather-2fspb\" (UID: \"c4308aa5-f445-46c1-9103-80d42b64ce59\") " pod="openshift-must-gather-zwm6c/must-gather-2fspb" Oct 30 05:38:43.692348 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.692325 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5md92\" (UniqueName: \"kubernetes.io/projected/c4308aa5-f445-46c1-9103-80d42b64ce59-kube-api-access-5md92\") pod \"must-gather-2fspb\" (UID: \"c4308aa5-f445-46c1-9103-80d42b64ce59\") " pod="openshift-must-gather-zwm6c/must-gather-2fspb" Oct 30 05:38:43.772780 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:43.772727 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zwm6c/must-gather-2fspb" Oct 30 05:38:44.074728 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:44.074700 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zwm6c/must-gather-2fspb"] Oct 30 05:38:44.076715 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:38:44.076687 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4308aa5_f445_46c1_9103_80d42b64ce59.slice/crio-bde1d1794a00f6e4bf5c659c79719a84ccba3024345fee4b04baa115cf1a30a5 WatchSource:0}: Error finding container bde1d1794a00f6e4bf5c659c79719a84ccba3024345fee4b04baa115cf1a30a5: Status 404 returned error can't find the container with id bde1d1794a00f6e4bf5c659c79719a84ccba3024345fee4b04baa115cf1a30a5 Oct 30 05:38:44.078559 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:44.078542 4793 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:38:44.530601 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:44.530565 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwm6c/must-gather-2fspb" event={"ID":"c4308aa5-f445-46c1-9103-80d42b64ce59","Type":"ContainerStarted","Data":"bde1d1794a00f6e4bf5c659c79719a84ccba3024345fee4b04baa115cf1a30a5"} Oct 30 05:38:45.257560 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:45.257363 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:38:48.097218 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:48.097180 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p2mmh_93cbeaeb-857f-4c3f-825f-62c46320edd0/global-pull-secret-syncer/0.log" Oct 30 05:38:48.699292 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:48.699246 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" Oct 30 05:38:49.351492 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:49.351391 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-177.ec2.internal_552cd6bdf9ef61710ec1ae6f1eae6b32/haproxy/0.log" Oct 30 05:38:49.565652 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:49.565616 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwm6c/must-gather-2fspb" event={"ID":"c4308aa5-f445-46c1-9103-80d42b64ce59","Type":"ContainerStarted","Data":"0d11c796701d6eb8f4b13346d97eff38a7e6837c09ed01c024f7354e886df56d"} Oct 30 05:38:49.565864 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:49.565660 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zwm6c/must-gather-2fspb" event={"ID":"c4308aa5-f445-46c1-9103-80d42b64ce59","Type":"ContainerStarted","Data":"93aae035048b88041a47e00afbc1b15b0200d5a2592f6eb636791a201a0327f7"} Oct 30 05:38:49.605058 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:49.604873 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zwm6c/must-gather-2fspb" podStartSLOduration=1.626445576 podStartE2EDuration="6.604856477s" totalImagesPullingTime="4.978410901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:38:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:49.604821926 +0000 UTC m=+6052.890696720" watchObservedRunningTime="2026-10-30 05:38:49.604856477 +0000 UTC m=+6052.890731254" Oct 30 05:38:50.993515 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:50.993477 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79d77bcfdb-6hwnl_a736cf03-a95d-4e46-9358-7b8e6bba483b/manager/0.log" Oct 30 05:38:51.481688 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:51.481653 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79d77bcfdb-6hwnl_a736cf03-a95d-4e46-9358-7b8e6bba483b/manager/0.log" Oct 30 05:38:54.150248 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:54.150193 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pxxpk_66328127-fc0b-4f8a-9599-2a82c297712d/manager/0.log" Oct 30 05:38:54.732648 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:54.732609 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pxxpk_66328127-fc0b-4f8a-9599-2a82c297712d/manager/0.log" Oct 30 05:38:55.726150 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:55.726119 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79d77bcfdb-6hwnl_a736cf03-a95d-4e46-9358-7b8e6bba483b/manager/0.log" Oct 30 05:38:56.074348 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:56.074271 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79d77bcfdb-6hwnl_a736cf03-a95d-4e46-9358-7b8e6bba483b/manager/0.log" Oct 30 05:38:58.413632 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:58.413599 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-c4zkg_8af2e200-3999-4f9b-a10c-1407bef727e3/manager/0.log" Oct 30 05:38:58.564136 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:58.564093 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pg5vv_12b6686c-fc28-4cc0-9c53-252a4e8a02bd/manager/0.log" Oct 30 05:38:58.899405 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:58.899328 4793 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2"] Oct 30 05:38:58.906741 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:58.906713 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:58.919255 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:58.919231 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2"] Oct 30 05:38:58.933598 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:58.928111 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xfkr\"/\"openshift-service-ca.crt\"" Oct 30 05:38:58.933598 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:58.928171 4793 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9xfkr\"/\"kube-root-ca.crt\"" Oct 30 05:38:58.933598 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:58.931805 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7nhv2_7975f468-e137-4d99-8a6d-ad9ac46c8961/manager/0.log" Oct 30 05:38:58.933598 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:58.932915 4793 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9xfkr\"/\"default-dockercfg-x74pv\"" Oct 30 05:38:59.014668 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.014637 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e7e277a-4a3f-4d76-9cc5-b4784b825997-podres\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.014940 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.014917 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e7e277a-4a3f-4d76-9cc5-b4784b825997-proc\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.015358 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.015086 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e7e277a-4a3f-4d76-9cc5-b4784b825997-lib-modules\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.015358 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.015137 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pjsg\" (UniqueName: \"kubernetes.io/projected/6e7e277a-4a3f-4d76-9cc5-b4784b825997-kube-api-access-9pjsg\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.015358 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.015206 4793 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e7e277a-4a3f-4d76-9cc5-b4784b825997-sys\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.115828 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.115788 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e7e277a-4a3f-4d76-9cc5-b4784b825997-podres\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.116078 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.115879 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e7e277a-4a3f-4d76-9cc5-b4784b825997-proc\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.116078 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.115897 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6e7e277a-4a3f-4d76-9cc5-b4784b825997-podres\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.116078 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.115913 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e7e277a-4a3f-4d76-9cc5-b4784b825997-lib-modules\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.116078 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.115970 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6e7e277a-4a3f-4d76-9cc5-b4784b825997-proc\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.116078 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.115987 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6e7e277a-4a3f-4d76-9cc5-b4784b825997-lib-modules\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.116078 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.115987 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pjsg\" (UniqueName: \"kubernetes.io/projected/6e7e277a-4a3f-4d76-9cc5-b4784b825997-kube-api-access-9pjsg\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.116078 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.116039 4793 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e7e277a-4a3f-4d76-9cc5-b4784b825997-sys\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.116625 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.116171 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e7e277a-4a3f-4d76-9cc5-b4784b825997-sys\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.140447 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.140388 4793 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pjsg\" (UniqueName: \"kubernetes.io/projected/6e7e277a-4a3f-4d76-9cc5-b4784b825997-kube-api-access-9pjsg\") pod \"perf-node-gather-daemonset-rc7z2\" (UID: \"6e7e277a-4a3f-4d76-9cc5-b4784b825997\") " pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.227755 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.227722 4793 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:38:59.542483 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.542450 4793 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2"] Oct 30 05:38:59.554162 ip-10-0-138-177 kubenswrapper[4793]: W1030 05:38:59.544837 4793 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6e7e277a_4a3f_4d76_9cc5_b4784b825997.slice/crio-34bef16a675a141c000f61b501f2cb1ad26f518fa9ae88ac965661633d6b7ba1 WatchSource:0}: Error finding container 34bef16a675a141c000f61b501f2cb1ad26f518fa9ae88ac965661633d6b7ba1: Status 404 returned error can't find the container with id 34bef16a675a141c000f61b501f2cb1ad26f518fa9ae88ac965661633d6b7ba1 Oct 30 05:38:59.634151 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:38:59.634075 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" event={"ID":"6e7e277a-4a3f-4d76-9cc5-b4784b825997","Type":"ContainerStarted","Data":"34bef16a675a141c000f61b501f2cb1ad26f518fa9ae88ac965661633d6b7ba1"} Oct 30 05:39:00.011601 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:00.011550 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pxxpk_66328127-fc0b-4f8a-9599-2a82c297712d/manager/0.log" Oct 30 05:39:00.016920 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:00.016890 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pxxpk_66328127-fc0b-4f8a-9599-2a82c297712d/manager/0.log" Oct 30 05:39:00.504350 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:00.504321 4793 scope.go:122] "RemoveContainer" containerID="09e911a795671ecd4caeaa94966b5beae96777be0b6e98d9eab4881ee1b03408" Oct 30 05:39:00.513397 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:00.513376 4793 scope.go:122] "RemoveContainer" containerID="6ca6f3bafec6c4bbdff0357928f12879e6d7d013d4e3e8132c2f4f0c49844b41" Oct 30 05:39:00.642427 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:00.642379 4793 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" event={"ID":"6e7e277a-4a3f-4d76-9cc5-b4784b825997","Type":"ContainerStarted","Data":"7aa73358f74e93d6089e5d28759ee745e62445cb4b222e16ec3948b26c0e9874"} Oct 30 05:39:00.642876 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:00.642549 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:39:00.683050 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:00.682995 4793 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" podStartSLOduration=2.682978496 podStartE2EDuration="2.682978496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:39:00.682500751 +0000 UTC m=+6063.968375552" watchObservedRunningTime="2026-10-30 05:39:00.682978496 +0000 UTC m=+6063.968853272" Oct 30 05:39:01.481477 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:01.481440 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79d77bcfdb-6hwnl_a736cf03-a95d-4e46-9358-7b8e6bba483b/manager/0.log" Oct 30 05:39:01.483218 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:01.483183 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-79d77bcfdb-6hwnl_a736cf03-a95d-4e46-9358-7b8e6bba483b/manager/0.log" Oct 30 05:39:02.315066 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:02.315026 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d8rtc_703f4255-11e5-4a34-a7d2-c1e26ef1fe3d/node-exporter/0.log" Oct 30 05:39:02.436173 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:02.436146 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d8rtc_703f4255-11e5-4a34-a7d2-c1e26ef1fe3d/kube-rbac-proxy/0.log" Oct 30 05:39:02.556982 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:02.556930 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d8rtc_703f4255-11e5-4a34-a7d2-c1e26ef1fe3d/init-textfile/0.log" Oct 30 05:39:02.784908 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:02.784857 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:39:03.892638 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:03.892602 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4075a738-d882-4257-9997-593dfc82570d/prometheus/0.log" Oct 30 05:39:04.016464 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:04.016404 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4075a738-d882-4257-9997-593dfc82570d/config-reloader/0.log" Oct 30 05:39:04.139090 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:04.139045 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4075a738-d882-4257-9997-593dfc82570d/thanos-sidecar/0.log" Oct 30 05:39:04.258994 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:04.258967 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4075a738-d882-4257-9997-593dfc82570d/kube-rbac-proxy-web/0.log" Oct 30 05:39:04.381856 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:04.381825 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4075a738-d882-4257-9997-593dfc82570d/kube-rbac-proxy/0.log" Oct 30 05:39:04.503161 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:04.503130 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4075a738-d882-4257-9997-593dfc82570d/kube-rbac-proxy-thanos/0.log" Oct 30 05:39:04.625323 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:04.625255 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4075a738-d882-4257-9997-593dfc82570d/init-config-reloader/0.log" Oct 30 05:39:05.313177 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:05.313144 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pxxpk_66328127-fc0b-4f8a-9599-2a82c297712d/manager/0.log" Oct 30 05:39:05.324620 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:05.324596 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-pxxpk_66328127-fc0b-4f8a-9599-2a82c297712d/manager/0.log" Oct 30 05:39:06.658196 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:06.658168 4793 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9xfkr/perf-node-gather-daemonset-rc7z2" Oct 30 05:39:07.370011 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:07.369978 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2_1179efa6-bb34-43ae-ab5d-ae9e3ec68028/istio-proxy/0.log" Oct 30 05:39:07.459059 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:07.459025 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2_1179efa6-bb34-43ae-ab5d-ae9e3ec68028/istio-proxy/0.log" Oct 30 05:39:07.631092 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:07.631016 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-c99g8_9b3ff486-e88f-459b-a0b1-aa5d88c6660d/istio-proxy/0.log" Oct 30 05:39:07.721363 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:07.721330 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-c99g8_9b3ff486-e88f-459b-a0b1-aa5d88c6660d/istio-proxy/0.log" Oct 30 05:39:07.825261 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:07.825235 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-h7nkg_84c295aa-6250-4bb0-bb4d-4fe24004a9f4/istio-proxy/0.log" Oct 30 05:39:07.920012 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:07.919931 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-h7nkg_84c295aa-6250-4bb0-bb4d-4fe24004a9f4/istio-proxy/0.log" Oct 30 05:39:08.023523 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:08.023479 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-mfjwn_4a556af0-394a-4c79-a3b8-971c207edf59/istio-proxy/0.log" Oct 30 05:39:08.113112 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:08.113088 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-mfjwn_4a556af0-394a-4c79-a3b8-971c207edf59/istio-proxy/0.log" Oct 30 05:39:08.310830 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:08.310799 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66_990707cc-7db4-490b-a520-9af03daeeeff/istio-proxy/0.log" Oct 30 05:39:08.390958 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:08.390922 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66_990707cc-7db4-490b-a520-9af03daeeeff/istio-proxy/0.log" Oct 30 05:39:08.444490 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:08.444463 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5688bd49d7-w75hb_e4815527-3e97-4f0b-bd06-be6277725976/payload-processing/0.log" Oct 30 05:39:08.520332 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:08.520308 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5688bd49d7-w75hb_e4815527-3e97-4f0b-bd06-be6277725976/payload-processing/0.log" Oct 30 05:39:10.633335 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:10.633251 4793 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gghj_cd9d674b-c3d9-4e78-bc55-81cedfc32881/konnectivity-agent/0.log" limit=1024 Oct 30 05:39:11.218808 ip-10-0-138-177 kubenswrapper[4793]: E1030 05:39:11.218769 4793 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b7bml" containerName="extractor" Oct 30 05:39:12.282109 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:12.282074 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fd5fcc55cqc8j2_1179efa6-bb34-43ae-ab5d-ae9e3ec68028/istio-proxy/0.log" Oct 30 05:39:12.585700 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:12.585623 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-c99g8_9b3ff486-e88f-459b-a0b1-aa5d88c6660d/istio-proxy/0.log" Oct 30 05:39:12.805013 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:12.804984 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-h7nkg_84c295aa-6250-4bb0-bb4d-4fe24004a9f4/istio-proxy/0.log" Oct 30 05:39:13.054482 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:13.054450 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-mfjwn_4a556af0-394a-4c79-a3b8-971c207edf59/istio-proxy/0.log" Oct 30 05:39:13.335179 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:13.335105 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7cc4b68895-sxj66_990707cc-7db4-490b-a520-9af03daeeeff/istio-proxy/0.log" Oct 30 05:39:13.472946 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:13.472909 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5688bd49d7-w75hb_e4815527-3e97-4f0b-bd06-be6277725976/payload-processing/0.log" Oct 30 05:39:14.928924 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:14.928891 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-c4zkg_8af2e200-3999-4f9b-a10c-1407bef727e3/manager/0.log" Oct 30 05:39:14.990776 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:14.990736 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pg5vv_12b6686c-fc28-4cc0-9c53-252a4e8a02bd/manager/0.log" Oct 30 05:39:15.049011 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:15.048981 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zrv7n_58706aa4-9bad-4e69-abdb-b9af8390f27c/dns/0.log" Oct 30 05:39:15.050323 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:15.050297 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-c4zkg_8af2e200-3999-4f9b-a10c-1407bef727e3/manager/0.log" Oct 30 05:39:15.105928 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:15.105891 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-pg5vv_12b6686c-fc28-4cc0-9c53-252a4e8a02bd/manager/0.log" Oct 30 05:39:15.157500 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:15.157473 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7nhv2_7975f468-e137-4d99-8a6d-ad9ac46c8961/manager/0.log" Oct 30 05:39:15.201708 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:15.201639 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zrv7n_58706aa4-9bad-4e69-abdb-b9af8390f27c/kube-rbac-proxy/0.log" Oct 30 05:39:15.321395 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:15.321366 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7nhv2_7975f468-e137-4d99-8a6d-ad9ac46c8961/manager/0.log" Oct 30 05:39:15.603171 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:15.603087 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zjzzn_6db45ae1-4ba3-466f-82f9-005ce3d82bd2/dns-node-resolver/0.log" Oct 30 05:39:17.355258 ip-10-0-138-177 kubenswrapper[4793]: I1030 05:39:17.355225 4793 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x58t9_73ae3560-963a-46a6-a052-21db48a98360/node-ca/0.log"